Jul 21 00:14:53.966185 ip-10-0-139-117 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 21 00:14:53.966197 ip-10-0-139-117 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 21 00:14:53.966203 ip-10-0-139-117 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 21 00:14:53.966417 ip-10-0-139-117 systemd[1]: Failed to start Kubernetes Kubelet. Jul 21 00:15:04.074555 ip-10-0-139-117 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 21 00:15:04.074578 ip-10-0-139-117 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 0611b7969ffa49f6bc3ab935b82d7a63 -- Jul 21 00:16:44.550569 ip-10-0-139-117 systemd[1]: Starting Kubernetes Kubelet... Jul 21 00:16:44.963795 ip-10-0-139-117 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 00:16:44.963795 ip-10-0-139-117 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 21 00:16:44.963795 ip-10-0-139-117 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 00:16:44.963795 ip-10-0-139-117 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 21 00:16:44.963795 ip-10-0-139-117 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 00:16:44.965716 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.965594 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 21 00:16:44.971010 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.970993 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 00:16:44.971010 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971010 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 00:16:44.971079 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971014 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 00:16:44.971079 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971018 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 00:16:44.971079 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971021 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 00:16:44.971079 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971024 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 00:16:44.971079 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971026 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 00:16:44.971079 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971029 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 00:16:44.971079 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971031 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 00:16:44.971079 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971035 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 00:16:44.971079 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971038 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 00:16:44.971079 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971040 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 00:16:44.971079 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971043 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 00:16:44.971079 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971046 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 00:16:44.971079 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971048 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 00:16:44.971079 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971050 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 00:16:44.971079 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971053 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 00:16:44.971079 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971056 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 21 00:16:44.971079 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971058 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 00:16:44.971079 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971062 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 00:16:44.971079 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971064 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 00:16:44.971079 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971067 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 00:16:44.971560 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971070 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 00:16:44.971560 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971074 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 00:16:44.971560 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971078 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 00:16:44.971560 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971081 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 00:16:44.971560 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971085 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 00:16:44.971560 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971088 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 00:16:44.971560 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971091 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 00:16:44.971560 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971094 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 00:16:44.971560 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971096 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 00:16:44.971560 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971100 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 00:16:44.971560 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971103 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 00:16:44.971560 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971105 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 00:16:44.971560 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971108 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 00:16:44.971560 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971110 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 00:16:44.971560 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971113 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 00:16:44.971560 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971116 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 00:16:44.971560 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971119 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 00:16:44.971560 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971121 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 00:16:44.971560 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971124 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 00:16:44.972105 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971126 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 00:16:44.972105 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971129 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 00:16:44.972105 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971131 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 00:16:44.972105 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971134 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 00:16:44.972105 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971136 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 00:16:44.972105 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971139 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 00:16:44.972105 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971141 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 00:16:44.972105 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971143 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 00:16:44.972105 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971146 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 00:16:44.972105 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971148 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 00:16:44.972105 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971151 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 00:16:44.972105 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971154 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 00:16:44.972105 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971156 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 00:16:44.972105 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971160 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 00:16:44.972105 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971164 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 00:16:44.972105 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971168 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 00:16:44.972105 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971171 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 00:16:44.972105 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971174 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 00:16:44.972105 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971176 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 00:16:44.972105 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971179 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 00:16:44.972604 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971181 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 00:16:44.972604 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971184 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 00:16:44.972604 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971186 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 00:16:44.972604 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971189 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 00:16:44.972604 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971192 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 00:16:44.972604 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971195 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 00:16:44.972604 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971198 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 00:16:44.972604 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971201 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 00:16:44.972604 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971203 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 00:16:44.972604 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971206 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 00:16:44.972604 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971209 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 00:16:44.972604 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971212 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 00:16:44.972604 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971214 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 00:16:44.972604 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971217 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 00:16:44.972604 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971219 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 00:16:44.972604 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971222 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 00:16:44.972604 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971224 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 00:16:44.972604 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971226 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 00:16:44.972604 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971229 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 00:16:44.972604 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971231 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 00:16:44.973078 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971234 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 00:16:44.973078 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971236 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 00:16:44.973078 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971238 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 00:16:44.973078 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971241 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 00:16:44.973078 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971243 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 00:16:44.973078 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971246 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 00:16:44.973078 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971250 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 00:16:44.973078 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971253 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 00:16:44.973078 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971255 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 00:16:44.973078 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971258 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 00:16:44.973078 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971260 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 00:16:44.973078 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971263 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 00:16:44.973078 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971265 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 00:16:44.973078 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971268 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 00:16:44.973078 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971270 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 00:16:44.973078 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971273 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 00:16:44.973078 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971275 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 00:16:44.973078 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.971279 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 00:16:44.973078 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972036 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 00:16:44.973576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972043 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 00:16:44.973576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972047 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 00:16:44.973576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972050 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 00:16:44.973576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972053 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 00:16:44.973576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972056 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 00:16:44.973576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972067 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 00:16:44.973576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972070 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 00:16:44.973576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972073 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 00:16:44.973576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972075 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 00:16:44.973576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972078 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 00:16:44.973576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972082 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 00:16:44.973576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972086 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 00:16:44.973576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972089 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 00:16:44.973576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972092 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 00:16:44.973576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972095 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 00:16:44.973576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972097 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 00:16:44.973576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972100 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 00:16:44.973576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972103 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 00:16:44.973576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972105 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 00:16:44.974041 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972108 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 00:16:44.974041 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972111 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 00:16:44.974041 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972114 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 00:16:44.974041 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972117 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 00:16:44.974041 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972120 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 00:16:44.974041 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972122 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 00:16:44.974041 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972125 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 00:16:44.974041 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972127 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 00:16:44.974041 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972131 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 00:16:44.974041 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972134 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 21 00:16:44.974041 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972137 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 00:16:44.974041 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972139 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 00:16:44.974041 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972142 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 00:16:44.974041 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972144 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 00:16:44.974041 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972146 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 00:16:44.974041 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972149 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 00:16:44.974041 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972151 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 00:16:44.974041 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972154 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 00:16:44.974041 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972162 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 00:16:44.974041 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972165 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 00:16:44.974576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972167 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 00:16:44.974576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972170 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 00:16:44.974576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972172 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 00:16:44.974576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972175 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 00:16:44.974576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972177 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 00:16:44.974576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972180 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 00:16:44.974576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972183 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 00:16:44.974576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972186 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 00:16:44.974576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972188 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 00:16:44.974576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972191 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 00:16:44.974576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972194 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 00:16:44.974576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972196 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 00:16:44.974576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972199 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 00:16:44.974576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972201 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 00:16:44.974576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972204 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 00:16:44.974576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972207 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 00:16:44.974576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972209 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 00:16:44.974576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972212 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 00:16:44.974576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972214 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 00:16:44.974576 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972217 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 00:16:44.975054 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972219 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 00:16:44.975054 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972221 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 00:16:44.975054 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972224 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 00:16:44.975054 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972226 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 00:16:44.975054 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972228 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 00:16:44.975054 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972231 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 00:16:44.975054 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972233 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 00:16:44.975054 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972235 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 00:16:44.975054 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972238 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 00:16:44.975054 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972240 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 00:16:44.975054 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972243 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 00:16:44.975054 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972253 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 00:16:44.975054 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972256 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 00:16:44.975054 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972258 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 00:16:44.975054 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972261 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 00:16:44.975054 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972263 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 00:16:44.975054 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972266 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 00:16:44.975054 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972268 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 00:16:44.975054 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972271 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 00:16:44.975054 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972273 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 00:16:44.975551 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972276 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 00:16:44.975551 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972278 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 00:16:44.975551 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972295 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 00:16:44.975551 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972298 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 00:16:44.975551 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972301 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 00:16:44.975551 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972304 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 00:16:44.975551 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972307 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 00:16:44.975551 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972309 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 00:16:44.975551 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972313 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 00:16:44.975551 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972316 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 00:16:44.975551 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972319 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 00:16:44.975551 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972322 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 00:16:44.975551 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972325 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 00:16:44.975551 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972328 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 00:16:44.975551 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972331 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 00:16:44.975551 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972334 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 00:16:44.975551 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972336 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 00:16:44.975551 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972339 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 00:16:44.975551 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972342 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 00:16:44.976017 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972742 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 00:16:44.976017 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972747 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 00:16:44.976017 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972750 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 00:16:44.976017 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972753 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 00:16:44.976017 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972756 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 00:16:44.976017 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972759 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 00:16:44.976017 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972762 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 00:16:44.976017 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972765 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 00:16:44.976017 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972768 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 00:16:44.976017 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972771 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 00:16:44.976017 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972773 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 00:16:44.976017 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972776 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 00:16:44.976017 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972778 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 00:16:44.976017 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972781 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 00:16:44.976017 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972783 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 00:16:44.976017 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972786 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 00:16:44.976017 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972789 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 00:16:44.976017 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972792 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 00:16:44.976017 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972794 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 00:16:44.976017 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972797 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 00:16:44.976508 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972799 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 00:16:44.976508 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972802 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 00:16:44.976508 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972804 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 00:16:44.976508 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972807 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 00:16:44.976508 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972809 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 00:16:44.976508 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972811 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 00:16:44.976508 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972814 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 00:16:44.976508 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972817 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 00:16:44.976508 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972819 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 00:16:44.976508 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972822 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 00:16:44.976508 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972824 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 00:16:44.976508 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972826 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 00:16:44.976508 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972829 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 00:16:44.976508 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972831 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 00:16:44.976508 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972833 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 00:16:44.976508 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972836 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 00:16:44.976508 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972838 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 00:16:44.976508 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972842 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 00:16:44.976508 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972844 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 00:16:44.976508 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972847 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 00:16:44.976979 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972849 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 00:16:44.976979 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972852 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 00:16:44.976979 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972854 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 00:16:44.976979 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972857 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 00:16:44.976979 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972859 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 00:16:44.976979 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972863 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 00:16:44.976979 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972865 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 00:16:44.976979 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972868 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 00:16:44.976979 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972870 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 00:16:44.976979 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972875 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 00:16:44.976979 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972877 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 00:16:44.976979 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972880 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 00:16:44.976979 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972882 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 00:16:44.976979 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972884 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 00:16:44.976979 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972887 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 00:16:44.976979 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972889 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 00:16:44.976979 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972892 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 00:16:44.976979 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972894 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 00:16:44.976979 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972896 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 00:16:44.976979 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972899 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 00:16:44.977463 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972901 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 00:16:44.977463 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972904 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 00:16:44.977463 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972906 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 00:16:44.977463 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972908 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 00:16:44.977463 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972911 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 00:16:44.977463 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972914 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 00:16:44.977463 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972916 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 00:16:44.977463 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972918 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 00:16:44.977463 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972921 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 00:16:44.977463 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972924 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 00:16:44.977463 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972927 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 00:16:44.977463 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972929 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 00:16:44.977463 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972931 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 00:16:44.977463 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972934 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 00:16:44.977463 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972936 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 00:16:44.977463 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972939 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 00:16:44.977463 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972942 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 00:16:44.977463 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972944 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 00:16:44.977463 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972946 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 00:16:44.977463 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972949 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 00:16:44.977948 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972951 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 00:16:44.977948 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972956 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 00:16:44.977948 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972958 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 00:16:44.977948 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972960 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 21 00:16:44.977948 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972963 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 00:16:44.977948 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972965 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 00:16:44.977948 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972967 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 00:16:44.977948 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972970 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 00:16:44.977948 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972972 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 00:16:44.977948 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972975 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 00:16:44.977948 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972979 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 00:16:44.977948 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972982 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 00:16:44.977948 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972985 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 00:16:44.977948 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972987 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 00:16:44.977948 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972991 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 00:16:44.977948 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972994 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 00:16:44.977948 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.972997 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 00:16:44.977948 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973002 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 00:16:44.977948 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973005 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 00:16:44.978437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973076 2580 flags.go:64] FLAG: --address="0.0.0.0" Jul 21 00:16:44.978437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973084 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 21 00:16:44.978437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973098 2580 flags.go:64] FLAG: --anonymous-auth="true" Jul 21 00:16:44.978437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973105 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 21 00:16:44.978437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973112 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 21 00:16:44.978437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973118 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 21 00:16:44.978437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973124 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 21 00:16:44.978437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973128 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 21 00:16:44.978437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973132 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 21 00:16:44.978437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973135 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 21 00:16:44.978437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973138 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 21 00:16:44.978437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973141 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 21 00:16:44.978437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973144 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 21 00:16:44.978437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973147 2580 flags.go:64] FLAG: --cgroup-root="" Jul 21 00:16:44.978437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973151 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 21 00:16:44.978437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973153 2580 flags.go:64] FLAG: --client-ca-file="" Jul 21 00:16:44.978437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973156 2580 flags.go:64] FLAG: --cloud-provider="external" Jul 21 00:16:44.978437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973159 2580 flags.go:64] FLAG: --cluster-dns="[]" Jul 21 00:16:44.978437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973165 2580 flags.go:64] FLAG: --cluster-domain="" Jul 21 00:16:44.978437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973167 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 21 00:16:44.978437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973170 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 21 00:16:44.978437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973173 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 21 00:16:44.978437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973176 2580 flags.go:64] FLAG: --container-log-max-files="5" Jul 21 00:16:44.978437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973180 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973183 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973186 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973189 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973192 2580 flags.go:64] FLAG: --contention-profiling="false" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973195 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973198 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973201 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973204 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973208 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973211 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973214 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973216 2580 flags.go:64] FLAG: --enable-load-reader="false" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973219 2580 flags.go:64] FLAG: --enable-server="true" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973223 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973227 2580 flags.go:64] FLAG: --event-burst="100" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973230 2580 flags.go:64] FLAG: --event-qps="50" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973233 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973235 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973238 2580 flags.go:64] FLAG: --eviction-hard="" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973242 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973244 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973247 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973250 2580 flags.go:64] FLAG: --eviction-soft="" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973253 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 21 00:16:44.979015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973256 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 21 00:16:44.979635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973259 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 21 00:16:44.979635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973261 2580 flags.go:64] FLAG: --experimental-mounter-path="" Jul 21 00:16:44.979635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973264 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 21 00:16:44.979635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973267 2580 flags.go:64] FLAG: --fail-swap-on="true" Jul 21 00:16:44.979635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973269 2580 flags.go:64] FLAG: --feature-gates="" Jul 21 00:16:44.979635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973273 2580 flags.go:64] FLAG: --file-check-frequency="20s" Jul 21 00:16:44.979635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973276 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 21 00:16:44.979635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973279 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 21 00:16:44.979635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973295 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 21 00:16:44.979635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973298 2580 flags.go:64] FLAG: --healthz-port="10248" Jul 21 00:16:44.979635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973301 2580 flags.go:64] FLAG: --help="false" Jul 21 00:16:44.979635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973304 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-139-117.ec2.internal" Jul 21 00:16:44.979635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973307 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 21 00:16:44.979635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973309 2580 flags.go:64] FLAG: --http-check-frequency="20s" Jul 21 00:16:44.979635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973312 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 21 00:16:44.979635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973315 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 21 00:16:44.979635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973319 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 21 00:16:44.979635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973322 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 21 00:16:44.979635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973324 2580 flags.go:64] FLAG: --image-service-endpoint="" Jul 21 00:16:44.979635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973327 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 21 00:16:44.979635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973330 2580 flags.go:64] FLAG: --kube-api-burst="100" Jul 21 00:16:44.979635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973334 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 21 00:16:44.979635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973337 2580 flags.go:64] FLAG: --kube-api-qps="50" Jul 21 00:16:44.979635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973340 2580 flags.go:64] FLAG: --kube-reserved="" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973343 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973346 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973348 2580 flags.go:64] FLAG: --kubelet-cgroups="" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973351 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973354 2580 flags.go:64] FLAG: --lock-file="" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973357 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973360 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973363 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973368 2580 flags.go:64] FLAG: --log-json-split-stream="false" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973371 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973374 2580 flags.go:64] FLAG: --log-text-split-stream="false" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973377 2580 flags.go:64] FLAG: --logging-format="text" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973380 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973383 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973386 2580 flags.go:64] FLAG: --manifest-url="" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973388 2580 flags.go:64] FLAG: --manifest-url-header="" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973393 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973396 2580 flags.go:64] FLAG: --max-open-files="1000000" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973400 2580 flags.go:64] FLAG: --max-pods="110" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973403 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973406 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973408 2580 flags.go:64] FLAG: --memory-manager-policy="None" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973411 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973414 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 21 00:16:44.980262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973417 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973420 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973430 2580 flags.go:64] FLAG: --node-status-max-images="50" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973433 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973436 2580 flags.go:64] FLAG: --oom-score-adj="-999" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973439 2580 flags.go:64] FLAG: --pod-cidr="" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973442 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973448 2580 flags.go:64] FLAG: --pod-manifest-path="" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973451 2580 flags.go:64] FLAG: --pod-max-pids="-1" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973454 2580 flags.go:64] FLAG: --pods-per-core="0" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973457 2580 flags.go:64] FLAG: --port="10250" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973459 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973462 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08ee04ea895554158" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973465 2580 flags.go:64] FLAG: --qos-reserved="" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973468 2580 flags.go:64] FLAG: --read-only-port="10255" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973471 2580 flags.go:64] FLAG: --register-node="true" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973473 2580 flags.go:64] FLAG: --register-with-taints="" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973477 2580 flags.go:64] FLAG: --registry-burst="10" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973480 2580 flags.go:64] FLAG: --registry-qps="5" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973483 2580 flags.go:64] FLAG: --reserved-cpus="" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973485 2580 flags.go:64] FLAG: --reserved-memory="" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973489 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973492 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973495 2580 flags.go:64] FLAG: --rotate-certificates="false" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973498 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 21 00:16:44.980895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973501 2580 flags.go:64] FLAG: --runonce="false" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973503 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973506 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973509 2580 flags.go:64] FLAG: --seccomp-default="false" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973512 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973516 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973519 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973522 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973525 2580 flags.go:64] FLAG: --storage-driver-password="root" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973527 2580 flags.go:64] FLAG: --storage-driver-secure="false" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973530 2580 flags.go:64] FLAG: --storage-driver-table="stats" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973533 2580 flags.go:64] FLAG: --storage-driver-user="root" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973535 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973538 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973541 2580 flags.go:64] FLAG: --system-cgroups="" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973544 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973549 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973551 2580 flags.go:64] FLAG: --tls-cert-file="" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973554 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973558 2580 flags.go:64] FLAG: --tls-min-version="" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973561 2580 flags.go:64] FLAG: --tls-private-key-file="" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973564 2580 flags.go:64] FLAG: --topology-manager-policy="none" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973566 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973569 2580 flags.go:64] FLAG: --topology-manager-scope="container" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973571 2580 flags.go:64] FLAG: --v="2" Jul 21 00:16:44.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973575 2580 flags.go:64] FLAG: --version="false" Jul 21 00:16:44.982120 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973580 2580 flags.go:64] FLAG: --vmodule="" Jul 21 00:16:44.982120 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973583 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 21 00:16:44.982120 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.973587 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 21 00:16:44.982120 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973704 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 00:16:44.982120 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973709 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 00:16:44.982120 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973712 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 00:16:44.982120 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973715 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 00:16:44.982120 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973718 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 00:16:44.982120 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973720 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 00:16:44.982120 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973722 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 00:16:44.982120 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973725 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 00:16:44.982120 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973730 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 00:16:44.982120 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973732 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 00:16:44.982120 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973735 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 00:16:44.982120 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973740 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 00:16:44.982120 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973743 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 00:16:44.982120 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973747 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 00:16:44.982120 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973751 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 00:16:44.982120 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973754 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 00:16:44.982120 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973757 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 00:16:44.982644 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973759 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 00:16:44.982644 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973762 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 00:16:44.982644 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973765 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 00:16:44.982644 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973767 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 00:16:44.982644 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973770 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 00:16:44.982644 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973772 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 00:16:44.982644 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973775 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 00:16:44.982644 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973778 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 00:16:44.982644 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973780 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 00:16:44.982644 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973782 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 00:16:44.982644 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973785 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 00:16:44.982644 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973787 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 00:16:44.982644 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973790 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 00:16:44.982644 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973792 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 00:16:44.982644 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973794 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 00:16:44.982644 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973797 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 00:16:44.982644 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973799 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 00:16:44.982644 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973802 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 00:16:44.982644 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973804 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 00:16:44.982644 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973807 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 00:16:44.983150 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973809 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 00:16:44.983150 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973811 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 00:16:44.983150 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973814 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 00:16:44.983150 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973819 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 00:16:44.983150 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973822 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 00:16:44.983150 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973825 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 00:16:44.983150 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973829 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 00:16:44.983150 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973831 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 00:16:44.983150 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973834 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 00:16:44.983150 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973837 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 00:16:44.983150 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973839 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 00:16:44.983150 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973842 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 00:16:44.983150 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973845 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 00:16:44.983150 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973847 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 00:16:44.983150 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973850 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 00:16:44.983150 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973853 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 00:16:44.983150 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973855 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 00:16:44.983150 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973858 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 00:16:44.983150 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973861 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 00:16:44.983639 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973863 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 00:16:44.983639 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973866 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 00:16:44.983639 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973869 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 00:16:44.983639 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973871 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 00:16:44.983639 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973874 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 00:16:44.983639 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973876 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 00:16:44.983639 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973878 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 00:16:44.983639 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973881 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 00:16:44.983639 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973883 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 00:16:44.983639 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973886 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 00:16:44.983639 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973888 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 00:16:44.983639 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973890 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 00:16:44.983639 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973894 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 00:16:44.983639 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973897 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 00:16:44.983639 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973900 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 00:16:44.983639 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973903 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 00:16:44.983639 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973906 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 21 00:16:44.983639 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973909 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 00:16:44.983639 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973911 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 00:16:44.984106 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973915 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 00:16:44.984106 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973917 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 00:16:44.984106 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973920 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 00:16:44.984106 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973922 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 00:16:44.984106 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973925 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 00:16:44.984106 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973927 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 00:16:44.984106 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973930 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 00:16:44.984106 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973932 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 00:16:44.984106 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973935 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 00:16:44.984106 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973937 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 00:16:44.984106 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973941 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 00:16:44.984106 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973943 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 00:16:44.984106 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973945 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 00:16:44.984106 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973948 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 00:16:44.984106 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973950 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 00:16:44.984106 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973952 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 00:16:44.984106 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973955 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 00:16:44.984106 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973957 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 00:16:44.984106 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973960 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 00:16:44.984106 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973962 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 00:16:44.984646 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973965 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 00:16:44.984646 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973967 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 00:16:44.984646 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973970 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 00:16:44.984646 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.973972 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 00:16:44.984646 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.974502 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 00:16:44.984646 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.981709 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 21 00:16:44.984646 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.981726 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 21 00:16:44.984646 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.983659 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 21 00:16:44.984646 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.983668 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 21 00:16:44.984646 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983733 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 00:16:44.984646 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983739 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 00:16:44.984646 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983742 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 00:16:44.984646 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983746 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 00:16:44.984646 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983749 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 00:16:44.984646 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983752 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 00:16:44.985018 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983754 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 00:16:44.985018 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983757 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 00:16:44.985018 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983760 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 00:16:44.985018 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983763 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 00:16:44.985018 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983766 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 00:16:44.985018 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983769 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 00:16:44.985018 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983771 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 00:16:44.985018 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983774 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 00:16:44.985018 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983777 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 00:16:44.985018 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983779 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 00:16:44.985018 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983782 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 00:16:44.985018 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983784 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 00:16:44.985018 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983787 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 00:16:44.985018 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983789 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 00:16:44.985018 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983792 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 00:16:44.985018 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983794 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 00:16:44.985018 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983796 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 00:16:44.985018 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983799 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 00:16:44.985018 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983802 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 00:16:44.985018 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983804 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 00:16:44.985503 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983807 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 00:16:44.985503 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983809 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 00:16:44.985503 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983812 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 00:16:44.985503 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983816 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 00:16:44.985503 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983818 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 00:16:44.985503 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983821 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 00:16:44.985503 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983823 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 00:16:44.985503 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983826 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 00:16:44.985503 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983828 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 00:16:44.985503 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983831 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 00:16:44.985503 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983833 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 00:16:44.985503 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983835 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 00:16:44.985503 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983838 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 00:16:44.985503 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983840 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 00:16:44.985503 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983842 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 00:16:44.985503 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983845 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 00:16:44.985503 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983849 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 00:16:44.985503 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983852 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 00:16:44.985503 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983855 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 00:16:44.985993 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983858 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 00:16:44.985993 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983862 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 00:16:44.985993 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983865 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 00:16:44.985993 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983867 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 00:16:44.985993 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983870 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 00:16:44.985993 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983872 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 00:16:44.985993 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983874 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 00:16:44.985993 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983877 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 00:16:44.985993 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983879 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 00:16:44.985993 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983881 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 00:16:44.985993 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983884 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 00:16:44.985993 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983886 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 00:16:44.985993 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983889 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 00:16:44.985993 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983891 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 00:16:44.985993 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983893 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 00:16:44.985993 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983896 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 00:16:44.985993 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983899 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 00:16:44.985993 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983902 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 00:16:44.985993 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983905 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 00:16:44.985993 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983907 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 00:16:44.986511 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983910 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 00:16:44.986511 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983912 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 00:16:44.986511 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983915 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 00:16:44.986511 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983918 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 00:16:44.986511 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983920 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 00:16:44.986511 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983922 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 00:16:44.986511 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983925 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 00:16:44.986511 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983927 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 00:16:44.986511 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983930 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 00:16:44.986511 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983932 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 00:16:44.986511 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983934 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 00:16:44.986511 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983936 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 00:16:44.986511 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983939 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 00:16:44.986511 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983941 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 00:16:44.986511 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983945 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 00:16:44.986511 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983948 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 00:16:44.986511 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983951 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 00:16:44.986511 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983953 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 00:16:44.986511 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983956 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 00:16:44.986977 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983958 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 00:16:44.986977 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983961 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 00:16:44.986977 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983963 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 00:16:44.986977 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983966 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 00:16:44.986977 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983968 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 00:16:44.986977 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983971 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 21 00:16:44.986977 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983973 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 00:16:44.986977 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983976 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 00:16:44.986977 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983979 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 00:16:44.986977 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983981 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 00:16:44.986977 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983984 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 00:16:44.986977 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983987 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 00:16:44.986977 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983989 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 00:16:44.986977 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983992 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 00:16:44.986977 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.983995 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 00:16:44.987377 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.984000 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 00:16:44.987377 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984114 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 00:16:44.987377 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984119 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 00:16:44.987377 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984122 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 00:16:44.987377 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984125 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 00:16:44.987377 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984128 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 00:16:44.987377 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984131 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 00:16:44.987377 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984134 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 00:16:44.987377 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984137 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 00:16:44.987377 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984139 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 00:16:44.987377 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984141 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 00:16:44.987377 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984144 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 00:16:44.987377 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984146 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 21 00:16:44.987377 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984149 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 00:16:44.987377 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984152 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 00:16:44.987746 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984154 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 00:16:44.987746 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984156 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 00:16:44.987746 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984159 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 00:16:44.987746 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984162 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 00:16:44.987746 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984164 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 00:16:44.987746 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984166 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 00:16:44.987746 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984169 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 00:16:44.987746 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984171 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 00:16:44.987746 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984174 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 00:16:44.987746 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984176 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 00:16:44.987746 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984179 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 00:16:44.987746 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984182 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 00:16:44.987746 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984185 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 00:16:44.987746 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984187 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 00:16:44.987746 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984190 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 00:16:44.987746 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984192 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 00:16:44.987746 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984195 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 00:16:44.987746 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984197 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 00:16:44.987746 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984200 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 00:16:44.987746 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984202 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 00:16:44.988227 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984204 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 00:16:44.988227 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984207 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 00:16:44.988227 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984209 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 00:16:44.988227 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984212 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 00:16:44.988227 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984214 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 00:16:44.988227 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984216 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 00:16:44.988227 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984219 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 00:16:44.988227 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984221 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 00:16:44.988227 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984223 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 00:16:44.988227 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984226 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 00:16:44.988227 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984228 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 00:16:44.988227 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984230 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 00:16:44.988227 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984233 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 00:16:44.988227 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984235 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 00:16:44.988227 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984237 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 00:16:44.988227 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984240 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 00:16:44.988227 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984242 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 00:16:44.988227 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984244 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 00:16:44.988227 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984247 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 00:16:44.988227 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984249 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 00:16:44.988805 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984251 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 00:16:44.988805 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984254 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 00:16:44.988805 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984256 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 00:16:44.988805 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984259 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 00:16:44.988805 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984262 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 00:16:44.988805 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984264 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 00:16:44.988805 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984267 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 00:16:44.988805 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984270 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 00:16:44.988805 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984272 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 00:16:44.988805 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984275 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 00:16:44.988805 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984277 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 00:16:44.988805 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984279 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 00:16:44.988805 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984296 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 00:16:44.988805 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984299 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 00:16:44.988805 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984301 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 00:16:44.988805 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984304 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 00:16:44.988805 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984307 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 00:16:44.988805 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984311 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 00:16:44.988805 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984314 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 00:16:44.989276 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984316 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 00:16:44.989276 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984319 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 00:16:44.989276 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984321 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 00:16:44.989276 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984324 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 00:16:44.989276 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984326 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 00:16:44.989276 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984328 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 00:16:44.989276 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984331 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 00:16:44.989276 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984333 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 00:16:44.989276 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984336 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 00:16:44.989276 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984338 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 00:16:44.989276 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984340 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 00:16:44.989276 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984343 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 00:16:44.989276 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984347 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 00:16:44.989276 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984350 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 00:16:44.989276 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984354 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 00:16:44.989276 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984357 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 00:16:44.989276 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984360 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 00:16:44.989276 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984363 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 00:16:44.989759 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984366 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 00:16:44.989759 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984368 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 00:16:44.989759 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984371 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 00:16:44.989759 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984374 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 00:16:44.989759 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984376 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 00:16:44.989759 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984379 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 00:16:44.989759 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984381 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 00:16:44.989759 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:44.984383 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 00:16:44.989759 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.984389 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 00:16:44.989759 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.984517 2580 server.go:962] "Client rotation is on, will bootstrap in background" Jul 21 00:16:44.989759 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.987017 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 21 00:16:44.989759 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.988550 2580 server.go:1019] "Starting client certificate rotation" Jul 21 00:16:44.989759 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.988652 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 00:16:44.989759 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.989452 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 00:16:44.994713 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.994657 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 00:16:44.998404 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:44.998381 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 00:16:45.012214 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.012188 2580 log.go:25] "Validated CRI v1 runtime API" Jul 21 00:16:45.014913 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.014893 2580 log.go:25] "Validated CRI v1 image API" Jul 21 00:16:45.016589 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.016574 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 21 00:16:45.019062 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.019041 2580 fs.go:135] Filesystem UUIDs: map[274dfc53-05b8-43bc-b5ec-72712cd0a9c4:/dev/nvme0n1p4 5f2a78eb-9ba1-4e94-b494-0cbf039aa628:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 21 00:16:45.019118 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.019062 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 21 00:16:45.024394 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.024261 2580 manager.go:216] Machine: {Timestamp:2026-07-21 00:16:45.023026865 +0000 UTC m=+0.352722464 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099382 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c547eca11df094fb921ac046b855f SystemUUID:ec2c547e-ca11-df09-4fb9-21ac046b855f BootID:0611b796-9ffa-49f6-bc3a-b935b82d7a63 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:35:69:c8:c1:4d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:35:69:c8:c1:4d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:a5:1d:4a:0a:7d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 21 00:16:45.024394 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.024391 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 21 00:16:45.024506 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.024480 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 21 00:16:45.025139 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.025125 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 00:16:45.027834 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.027806 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 21 00:16:45.027979 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.027837 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-117.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 21 00:16:45.028692 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.028682 2580 topology_manager.go:138] "Creating topology manager with none policy" Jul 21 00:16:45.028731 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.028695 2580 container_manager_linux.go:309] "Creating device plugin manager" Jul 21 00:16:45.028731 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.028702 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 00:16:45.029496 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.029484 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 00:16:45.031248 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.031236 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 21 00:16:45.035377 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.035364 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 21 00:16:45.035550 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.035540 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 21 00:16:45.035604 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.035570 2580 kubelet.go:486] "Attempting to sync node with API server" Jul 21 00:16:45.035604 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.035582 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 21 00:16:45.035604 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.035598 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 21 00:16:45.035724 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.035608 2580 kubelet.go:398] "Adding apiserver pod source" Jul 21 00:16:45.035724 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.035618 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 21 00:16:45.035993 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.035979 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 00:16:45.036044 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.036000 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 00:16:45.038327 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.038310 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 21 00:16:45.041532 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.041514 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 21 00:16:45.041610 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.041554 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 21 00:16:45.043016 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.043002 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 21 00:16:45.043116 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.043033 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 21 00:16:45.043116 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.043045 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 21 00:16:45.043116 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.043054 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 21 00:16:45.043116 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.043075 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 21 00:16:45.043116 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.043084 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 21 00:16:45.043116 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.043096 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 21 00:16:45.043116 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.043105 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 21 00:16:45.043116 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.043117 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 21 00:16:45.043395 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.043126 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 21 00:16:45.043395 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.043139 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 21 00:16:45.043395 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.043149 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 21 00:16:45.044213 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.044194 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 21 00:16:45.044267 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.044232 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 21 00:16:45.044650 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.044626 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gbv5t" Jul 21 00:16:45.045486 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:45.045462 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-117.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 21 00:16:45.045540 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:45.045465 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 21 00:16:45.047580 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.047567 2580 server.go:1268] "Started kubelet" Jul 21 00:16:45.047682 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.047655 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 21 00:16:45.047734 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.047658 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 21 00:16:45.047734 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.047706 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 21 00:16:45.048531 ip-10-0-139-117 systemd[1]: Started Kubernetes Kubelet. Jul 21 00:16:45.049683 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.049667 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gbv5t" Jul 21 00:16:45.049820 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.049798 2580 server.go:310] "Adding debug handlers to kubelet server" Jul 21 00:16:45.050106 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.050090 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 21 00:16:45.052584 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:45.052550 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 21 00:16:45.054198 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.054184 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-117.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 21 00:16:45.055325 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:45.054195 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-117.ec2.internal.18c425e2068158bb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-117.ec2.internal,UID:ip-10-0-139-117.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-117.ec2.internal,},FirstTimestamp:2026-07-21 00:16:45.047535803 +0000 UTC m=+0.377231402,LastTimestamp:2026-07-21 00:16:45.047535803 +0000 UTC m=+0.377231402,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-117.ec2.internal,}" Jul 21 00:16:45.061521 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.061491 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 21 00:16:45.061630 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.061496 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 21 00:16:45.061877 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:45.061858 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-117.ec2.internal\" not found" Jul 21 00:16:45.062391 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.062356 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 21 00:16:45.062391 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.062357 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 21 00:16:45.062391 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.062394 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 21 00:16:45.062605 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.062529 2580 reconstruct.go:97] "Volume reconstruction finished" Jul 21 00:16:45.062605 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.062538 2580 reconciler.go:29] "Reconciler: start to sync state" Jul 21 00:16:45.063326 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.063308 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 21 00:16:45.063326 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.063326 2580 factory.go:55] Registering systemd factory Jul 21 00:16:45.063430 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.063336 2580 factory.go:223] Registration of the systemd container factory successfully Jul 21 00:16:45.063567 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.063548 2580 factory.go:153] Registering CRI-O factory Jul 21 00:16:45.063567 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.063568 2580 factory.go:223] Registration of the crio container factory successfully Jul 21 00:16:45.063692 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.063587 2580 factory.go:103] Registering Raw factory Jul 21 00:16:45.063692 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.063597 2580 manager.go:1196] Started watching for new ooms in manager Jul 21 00:16:45.064329 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.064314 2580 manager.go:349] Starting recovery of all containers Jul 21 00:16:45.064421 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.064341 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 00:16:45.066942 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:45.066909 2580 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-117.ec2.internal\" not found" node="ip-10-0-139-117.ec2.internal" Jul 21 00:16:45.074903 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.074874 2580 manager.go:354] Recovery completed Jul 21 00:16:45.081250 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.081116 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 00:16:45.083520 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.083503 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-117.ec2.internal" event="NodeHasSufficientMemory" Jul 21 00:16:45.083585 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.083533 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-117.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 00:16:45.083585 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.083544 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-117.ec2.internal" event="NodeHasSufficientPID" Jul 21 00:16:45.084133 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.084119 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 21 00:16:45.084133 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.084130 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 21 00:16:45.084133 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.084147 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 21 00:16:45.084366 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.084132 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 00:16:45.084961 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.084948 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-117.ec2.internal" event="NodeHasSufficientMemory" Jul 21 00:16:45.085006 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.084975 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-117.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 00:16:45.085006 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.084988 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-117.ec2.internal" event="NodeHasSufficientPID" Jul 21 00:16:45.085554 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.085543 2580 policy_none.go:49] "None policy: Start" Jul 21 00:16:45.085597 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.085558 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 21 00:16:45.085597 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.085568 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 21 00:16:45.086802 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.086792 2580 policy_none.go:47] "Start" Jul 21 00:16:45.134406 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.121763 2580 manager.go:337] "Starting Device Plugin manager" Jul 21 00:16:45.134406 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:45.121796 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 21 00:16:45.134406 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.121810 2580 server.go:87] "Starting device plugin registration server" Jul 21 00:16:45.134406 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.122105 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 21 00:16:45.134406 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.122117 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 21 00:16:45.134406 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.122212 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 21 00:16:45.134406 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.122315 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 21 00:16:45.134406 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.122324 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 21 00:16:45.134406 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:45.122891 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 21 00:16:45.134406 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:45.122937 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-117.ec2.internal\" not found" Jul 21 00:16:45.174956 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.174922 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 21 00:16:45.176262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.176249 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 21 00:16:45.176343 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.176304 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 21 00:16:45.176343 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.176331 2580 kubelet.go:2448] "Starting kubelet main sync loop" Jul 21 00:16:45.176422 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:45.176363 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 21 00:16:45.178996 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.178976 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 00:16:45.222984 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.222907 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 00:16:45.224299 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.224265 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-117.ec2.internal" event="NodeHasSufficientMemory" Jul 21 00:16:45.224410 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.224314 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-117.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 00:16:45.224410 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.224325 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-117.ec2.internal" event="NodeHasSufficientPID" Jul 21 00:16:45.224410 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.224347 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-117.ec2.internal" Jul 21 00:16:45.232408 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.232388 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-117.ec2.internal" Jul 21 00:16:45.232497 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:45.232414 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-117.ec2.internal\": node \"ip-10-0-139-117.ec2.internal\" not found" Jul 21 00:16:45.244814 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:45.244788 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-117.ec2.internal\" not found" Jul 21 00:16:45.276766 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.276730 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-117.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-117.ec2.internal"] Jul 21 00:16:45.276885 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.276806 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 00:16:45.278594 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.278576 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-117.ec2.internal" event="NodeHasSufficientMemory" Jul 21 00:16:45.278667 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.278608 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-117.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 00:16:45.278667 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.278618 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-117.ec2.internal" event="NodeHasSufficientPID" Jul 21 00:16:45.280089 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.280076 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 00:16:45.280174 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.280161 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-117.ec2.internal" Jul 21 00:16:45.280209 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.280190 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 00:16:45.280801 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.280782 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-117.ec2.internal" event="NodeHasSufficientMemory" Jul 21 00:16:45.280914 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.280815 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-117.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 00:16:45.280914 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.280829 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-117.ec2.internal" event="NodeHasSufficientPID" Jul 21 00:16:45.280914 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.280900 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-117.ec2.internal" event="NodeHasSufficientMemory" Jul 21 00:16:45.281030 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.280924 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-117.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 00:16:45.281030 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.280938 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-117.ec2.internal" event="NodeHasSufficientPID" Jul 21 00:16:45.282450 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.282434 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-117.ec2.internal" Jul 21 00:16:45.282496 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.282465 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 00:16:45.283246 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.283230 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-117.ec2.internal" event="NodeHasSufficientMemory" Jul 21 00:16:45.283355 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.283251 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-117.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 00:16:45.283355 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.283261 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-117.ec2.internal" event="NodeHasSufficientPID" Jul 21 00:16:45.298799 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:45.298783 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-117.ec2.internal\" not found" node="ip-10-0-139-117.ec2.internal" Jul 21 00:16:45.303092 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:45.303078 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-117.ec2.internal\" not found" node="ip-10-0-139-117.ec2.internal" Jul 21 00:16:45.345164 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:45.345140 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-117.ec2.internal\" not found" Jul 21 00:16:45.363889 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.363866 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b00f410120824644017f2fc5f606062-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-117.ec2.internal\" (UID: \"0b00f410120824644017f2fc5f606062\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-117.ec2.internal" Jul 21 00:16:45.363987 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.363898 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/30cbc68e3783a788ccf4c7e2f8cc9298-config\") pod \"kube-apiserver-proxy-ip-10-0-139-117.ec2.internal\" (UID: \"30cbc68e3783a788ccf4c7e2f8cc9298\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-117.ec2.internal" Jul 21 00:16:45.363987 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.363916 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b00f410120824644017f2fc5f606062-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-117.ec2.internal\" (UID: \"0b00f410120824644017f2fc5f606062\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-117.ec2.internal" Jul 21 00:16:45.445328 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:45.445263 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-117.ec2.internal\" not found" Jul 21 00:16:45.464657 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.464632 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/30cbc68e3783a788ccf4c7e2f8cc9298-config\") pod \"kube-apiserver-proxy-ip-10-0-139-117.ec2.internal\" (UID: \"30cbc68e3783a788ccf4c7e2f8cc9298\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-117.ec2.internal" Jul 21 00:16:45.464657 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.464595 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/30cbc68e3783a788ccf4c7e2f8cc9298-config\") pod \"kube-apiserver-proxy-ip-10-0-139-117.ec2.internal\" (UID: \"30cbc68e3783a788ccf4c7e2f8cc9298\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-117.ec2.internal" Jul 21 00:16:45.464803 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.464689 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b00f410120824644017f2fc5f606062-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-117.ec2.internal\" (UID: \"0b00f410120824644017f2fc5f606062\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-117.ec2.internal" Jul 21 00:16:45.464803 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.464718 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0b00f410120824644017f2fc5f606062-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-117.ec2.internal\" (UID: \"0b00f410120824644017f2fc5f606062\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-117.ec2.internal" Jul 21 00:16:45.464803 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.464745 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b00f410120824644017f2fc5f606062-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-117.ec2.internal\" (UID: \"0b00f410120824644017f2fc5f606062\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-117.ec2.internal" Jul 21 00:16:45.464803 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.464759 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b00f410120824644017f2fc5f606062-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-117.ec2.internal\" (UID: \"0b00f410120824644017f2fc5f606062\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-117.ec2.internal" Jul 21 00:16:45.546137 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:45.546044 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-117.ec2.internal\" not found" Jul 21 00:16:45.603688 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.603656 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-117.ec2.internal" Jul 21 00:16:45.607089 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.607064 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-117.ec2.internal" Jul 21 00:16:45.646432 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:45.646401 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-117.ec2.internal\" not found" Jul 21 00:16:45.746943 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:45.746901 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-117.ec2.internal\" not found" Jul 21 00:16:45.847575 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:45.847479 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-117.ec2.internal\" not found" Jul 21 00:16:45.948029 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:45.947992 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-117.ec2.internal\" not found" Jul 21 00:16:45.987629 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.987596 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 21 00:16:45.988072 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.987749 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 00:16:45.988072 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:45.987786 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 00:16:46.048145 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:46.048107 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-117.ec2.internal\" not found" Jul 21 00:16:46.056237 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:46.056204 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 00:11:46 +0000 UTC" deadline="2028-03-16 22:23:11.513428953 +0000 UTC" Jul 21 00:16:46.056237 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:46.056231 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14518h6m25.457200896s" Jul 21 00:16:46.064392 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:46.064369 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 00:16:46.148661 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:46.148593 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-117.ec2.internal\" not found" Jul 21 00:16:46.190644 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:46.190592 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b00f410120824644017f2fc5f606062.slice/crio-16ee0c7dc74a9695c9b1551b0d332a46062c8f59b4a0e2e1b54cf729ccd3a56d WatchSource:0}: Error finding container 16ee0c7dc74a9695c9b1551b0d332a46062c8f59b4a0e2e1b54cf729ccd3a56d: Status 404 returned error can't find the container with id 16ee0c7dc74a9695c9b1551b0d332a46062c8f59b4a0e2e1b54cf729ccd3a56d Jul 21 00:16:46.199252 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:46.199234 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 00:16:46.205906 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:46.205874 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30cbc68e3783a788ccf4c7e2f8cc9298.slice/crio-f9daae0f7b9e3f5ded1b91e0aec50e55a21ec9d9274d78ee8eaae329be8e44cd WatchSource:0}: Error finding container f9daae0f7b9e3f5ded1b91e0aec50e55a21ec9d9274d78ee8eaae329be8e44cd: Status 404 returned error can't find the container with id f9daae0f7b9e3f5ded1b91e0aec50e55a21ec9d9274d78ee8eaae329be8e44cd Jul 21 00:16:46.249454 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:46.249415 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-117.ec2.internal\" not found" Jul 21 00:16:46.349936 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:46.349900 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-117.ec2.internal\" not found" Jul 21 00:16:46.450530 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:46.450451 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-117.ec2.internal\" not found" Jul 21 00:16:47.665647 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:47.665610 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-117.ec2.internal\" not found" Jul 21 00:16:47.756892 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:47.756862 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 00:16:47.776146 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:47.776118 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-117.ec2.internal" Jul 21 00:16:47.786236 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:47.786214 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 00:16:47.787364 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:47.787346 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-117.ec2.internal" Jul 21 00:16:47.795772 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:47.795745 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 00:16:48.151204 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.151172 2580 apiserver.go:52] "Watching apiserver" Jul 21 00:16:48.158872 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.158841 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 21 00:16:48.159337 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.159310 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-jlx72","openshift-cluster-node-tuning-operator/tuned-fcl75","openshift-dns/node-resolver-szxbf","openshift-multus/multus-additional-cni-plugins-k2vj4","openshift-multus/multus-scmjq","openshift-network-diagnostics/network-check-target-98dgh","openshift-ovn-kubernetes/ovnkube-node-mzgzs","kube-system/global-pull-secret-syncer-2psfz","kube-system/kube-apiserver-proxy-ip-10-0-139-117.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8","openshift-image-registry/node-ca-p76tl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-117.ec2.internal","openshift-multus/network-metrics-daemon-n8sh6","openshift-network-operator/iptables-alerter-6pvbf"] Jul 21 00:16:48.161831 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.161352 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.163977 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.163473 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:16:48.163977 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.163485 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 21 00:16:48.163977 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:48.163577 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8sh6" podUID="982c215c-29b3-45ac-8525-03c61f27e931" Jul 21 00:16:48.163977 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.163716 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 21 00:16:48.163977 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.163809 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 21 00:16:48.163977 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.163865 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m4s68\"" Jul 21 00:16:48.164680 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.164659 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 21 00:16:48.164772 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.164693 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 21 00:16:48.165673 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.165647 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-szxbf" Jul 21 00:16:48.166282 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.166267 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 21 00:16:48.167079 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.167059 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.168471 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.167727 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jwmlj\"" Jul 21 00:16:48.168471 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.168046 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 21 00:16:48.168471 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.168224 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 21 00:16:48.168917 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.168896 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.169584 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.169563 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 21 00:16:48.169879 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.169862 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pqj8k\"" Jul 21 00:16:48.170086 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.170069 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 21 00:16:48.170272 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.170255 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 21 00:16:48.170708 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.170683 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 21 00:16:48.170789 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.170753 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 21 00:16:48.171618 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.171213 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 21 00:16:48.171618 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.171421 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-h49pd\"" Jul 21 00:16:48.171801 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.171772 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:16:48.171893 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:48.171836 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98dgh" podUID="e6e90e27-ad89-49df-904c-d28dd5ecaea4" Jul 21 00:16:48.173505 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.173475 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6pvbf" Jul 21 00:16:48.175347 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.175328 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jlx72" Jul 21 00:16:48.175691 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.175674 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 21 00:16:48.175777 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.175682 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 21 00:16:48.175838 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.175773 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pbhtq\"" Jul 21 00:16:48.175918 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.175898 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 21 00:16:48.176042 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.176026 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 21 00:16:48.176666 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.176649 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.177991 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.177967 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 21 00:16:48.178083 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.178023 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-22jnk\"" Jul 21 00:16:48.178819 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.178803 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 21 00:16:48.179950 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.179930 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 21 00:16:48.180448 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.180408 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cnhz2\"" Jul 21 00:16:48.180633 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.180616 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 21 00:16:48.182642 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.182621 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" Jul 21 00:16:48.184438 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.184072 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p76tl" Jul 21 00:16:48.184650 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.184635 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 21 00:16:48.184827 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.184812 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 21 00:16:48.184902 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.184842 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 21 00:16:48.185003 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.184960 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8hjml\"" Jul 21 00:16:48.185883 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.185853 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:16:48.185962 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:48.185922 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2psfz" podUID="23335158-224d-46e1-b617-10ffa4acb460" Jul 21 00:16:48.186239 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.186221 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 21 00:16:48.187262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.187242 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ncz79\"" Jul 21 00:16:48.187402 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.187371 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 21 00:16:48.187484 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.187466 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 21 00:16:48.188585 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.188564 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 00:16:48.189042 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.188938 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-multus-cni-dir\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.189042 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.188972 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48d1f8d1-615b-4b43-ab84-f2b22e43a382-sys-fs\") pod \"aws-ebs-csi-driver-node-rnjq8\" (UID: \"48d1f8d1-615b-4b43-ab84-f2b22e43a382\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" Jul 21 00:16:48.189042 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.188992 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-host-slash\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.189042 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189027 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5355db2a-b7df-48d3-9554-b5eaa4e937ab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k2vj4\" (UID: \"5355db2a-b7df-48d3-9554-b5eaa4e937ab\") " pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.189351 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189060 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-etc-sysctl-d\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.189351 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189081 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-run-systemd\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.189351 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189105 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-host-run-ovn-kubernetes\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.189351 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189144 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xffqs\" (UniqueName: \"kubernetes.io/projected/982c215c-29b3-45ac-8525-03c61f27e931-kube-api-access-xffqs\") pod \"network-metrics-daemon-n8sh6\" (UID: \"982c215c-29b3-45ac-8525-03c61f27e931\") " pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:16:48.189351 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189178 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5355db2a-b7df-48d3-9554-b5eaa4e937ab-cni-binary-copy\") pod \"multus-additional-cni-plugins-k2vj4\" (UID: \"5355db2a-b7df-48d3-9554-b5eaa4e937ab\") " pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.189351 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189214 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sklg6\" (UniqueName: \"kubernetes.io/projected/00221678-4e87-4266-9c77-51182bb87c2b-kube-api-access-sklg6\") pod \"iptables-alerter-6pvbf\" (UID: \"00221678-4e87-4266-9c77-51182bb87c2b\") " pod="openshift-network-operator/iptables-alerter-6pvbf" Jul 21 00:16:48.189351 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189237 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m8vd\" (UniqueName: \"kubernetes.io/projected/48d1f8d1-615b-4b43-ab84-f2b22e43a382-kube-api-access-2m8vd\") pod \"aws-ebs-csi-driver-node-rnjq8\" (UID: \"48d1f8d1-615b-4b43-ab84-f2b22e43a382\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" Jul 21 00:16:48.189351 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189264 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-var-lib-openvswitch\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.189351 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189308 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-hostroot\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.189351 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189339 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5355db2a-b7df-48d3-9554-b5eaa4e937ab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k2vj4\" (UID: \"5355db2a-b7df-48d3-9554-b5eaa4e937ab\") " pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.189859 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189366 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-run-openvswitch\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.189859 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189392 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-ovnkube-config\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.189859 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189433 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48d1f8d1-615b-4b43-ab84-f2b22e43a382-registration-dir\") pod \"aws-ebs-csi-driver-node-rnjq8\" (UID: \"48d1f8d1-615b-4b43-ab84-f2b22e43a382\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" Jul 21 00:16:48.189859 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189463 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-multus-socket-dir-parent\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.189859 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189523 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-host-var-lib-cni-bin\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.189859 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189550 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-host-run-multus-certs\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.189859 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189592 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-var-lib-kubelet\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.189859 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189633 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-etc-sysctl-conf\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.189859 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189655 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-lib-modules\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.189859 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189690 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-host\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.189859 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189721 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48d1f8d1-615b-4b43-ab84-f2b22e43a382-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnjq8\" (UID: \"48d1f8d1-615b-4b43-ab84-f2b22e43a382\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" Jul 21 00:16:48.189859 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189777 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-host-run-netns\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.189859 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189799 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6c602047-2c64-4c38-ba0d-8042a77f4e24-etc-tuned\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.189859 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189820 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c602047-2c64-4c38-ba0d-8042a77f4e24-tmp\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.189859 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189844 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48d1f8d1-615b-4b43-ab84-f2b22e43a382-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnjq8\" (UID: \"48d1f8d1-615b-4b43-ab84-f2b22e43a382\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" Jul 21 00:16:48.189859 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189865 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-node-log\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.190580 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189901 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-run\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.190580 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189939 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/00221678-4e87-4266-9c77-51182bb87c2b-iptables-alerter-script\") pod \"iptables-alerter-6pvbf\" (UID: \"00221678-4e87-4266-9c77-51182bb87c2b\") " pod="openshift-network-operator/iptables-alerter-6pvbf" Jul 21 00:16:48.190580 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189966 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-host-var-lib-cni-multus\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.190580 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.189997 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-etc-kubernetes\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.190580 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190023 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-etc-openvswitch\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.190580 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190048 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs\") pod \"network-metrics-daemon-n8sh6\" (UID: \"982c215c-29b3-45ac-8525-03c61f27e931\") " pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:16:48.190580 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190072 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7lfv\" (UniqueName: \"kubernetes.io/projected/6c602047-2c64-4c38-ba0d-8042a77f4e24-kube-api-access-l7lfv\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.190580 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190096 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37628504-b773-4dad-a955-2a87d40ab71c-cni-binary-copy\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.190580 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190145 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-host-var-lib-kubelet\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.190580 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190172 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-env-overrides\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.190580 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190216 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-ovnkube-script-lib\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.190580 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190242 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5355db2a-b7df-48d3-9554-b5eaa4e937ab-system-cni-dir\") pod \"multus-additional-cni-plugins-k2vj4\" (UID: \"5355db2a-b7df-48d3-9554-b5eaa4e937ab\") " pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.190580 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190266 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-etc-kubernetes\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.190580 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190291 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-host-cni-netd\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.190580 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190316 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.190580 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190340 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r76wr\" (UniqueName: \"kubernetes.io/projected/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-kube-api-access-r76wr\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.191417 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190363 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/583c4be0-d3e2-4a43-b91c-d6a8903569da-hosts-file\") pod \"node-resolver-szxbf\" (UID: \"583c4be0-d3e2-4a43-b91c-d6a8903569da\") " pod="openshift-dns/node-resolver-szxbf" Jul 21 00:16:48.191417 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190394 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5355db2a-b7df-48d3-9554-b5eaa4e937ab-os-release\") pod \"multus-additional-cni-plugins-k2vj4\" (UID: \"5355db2a-b7df-48d3-9554-b5eaa4e937ab\") " pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.191417 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190421 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-etc-systemd\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.191417 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190444 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-sys\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.191417 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190465 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-host-run-k8s-cni-cncf-io\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.191417 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190523 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-host-kubelet\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.191417 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190549 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-ovn-node-metrics-cert\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.191417 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190572 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-etc-sysconfig\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.191417 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190595 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583c4be0-d3e2-4a43-b91c-d6a8903569da-tmp-dir\") pod \"node-resolver-szxbf\" (UID: \"583c4be0-d3e2-4a43-b91c-d6a8903569da\") " pod="openshift-dns/node-resolver-szxbf" Jul 21 00:16:48.191417 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190637 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48d1f8d1-615b-4b43-ab84-f2b22e43a382-socket-dir\") pod \"aws-ebs-csi-driver-node-rnjq8\" (UID: \"48d1f8d1-615b-4b43-ab84-f2b22e43a382\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" Jul 21 00:16:48.191417 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190662 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-multus-conf-dir\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.191417 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190687 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48d1f8d1-615b-4b43-ab84-f2b22e43a382-device-dir\") pod \"aws-ebs-csi-driver-node-rnjq8\" (UID: \"48d1f8d1-615b-4b43-ab84-f2b22e43a382\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" Jul 21 00:16:48.191417 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190712 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-etc-modprobe-d\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.191417 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190736 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-system-cni-dir\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.191417 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190774 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-cnibin\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.191417 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190799 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhcws\" (UniqueName: \"kubernetes.io/projected/37628504-b773-4dad-a955-2a87d40ab71c-kube-api-access-dhcws\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.191417 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190823 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-log-socket\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.192102 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190850 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-host-cni-bin\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.192102 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190917 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5355db2a-b7df-48d3-9554-b5eaa4e937ab-cnibin\") pod \"multus-additional-cni-plugins-k2vj4\" (UID: \"5355db2a-b7df-48d3-9554-b5eaa4e937ab\") " pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.192102 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190956 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zp89\" (UniqueName: \"kubernetes.io/projected/5355db2a-b7df-48d3-9554-b5eaa4e937ab-kube-api-access-8zp89\") pod \"multus-additional-cni-plugins-k2vj4\" (UID: \"5355db2a-b7df-48d3-9554-b5eaa4e937ab\") " pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.192102 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.190988 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnbcl\" (UniqueName: \"kubernetes.io/projected/e6e90e27-ad89-49df-904c-d28dd5ecaea4-kube-api-access-tnbcl\") pod \"network-check-target-98dgh\" (UID: \"e6e90e27-ad89-49df-904c-d28dd5ecaea4\") " pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:16:48.192102 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.191039 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-os-release\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.192102 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.191101 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-host-run-netns\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.192102 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.191120 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/00221678-4e87-4266-9c77-51182bb87c2b-host-slash\") pod \"iptables-alerter-6pvbf\" (UID: \"00221678-4e87-4266-9c77-51182bb87c2b\") " pod="openshift-network-operator/iptables-alerter-6pvbf" Jul 21 00:16:48.192102 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.191147 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/da6931b3-d9b0-4c63-a0ba-7eca03b046db-agent-certs\") pod \"konnectivity-agent-jlx72\" (UID: \"da6931b3-d9b0-4c63-a0ba-7eca03b046db\") " pod="kube-system/konnectivity-agent-jlx72" Jul 21 00:16:48.192102 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.191171 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-systemd-units\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.192102 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.191200 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-run-ovn\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.192102 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.191224 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5b7f\" (UniqueName: \"kubernetes.io/projected/583c4be0-d3e2-4a43-b91c-d6a8903569da-kube-api-access-j5b7f\") pod \"node-resolver-szxbf\" (UID: \"583c4be0-d3e2-4a43-b91c-d6a8903569da\") " pod="openshift-dns/node-resolver-szxbf" Jul 21 00:16:48.192102 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.191247 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37628504-b773-4dad-a955-2a87d40ab71c-multus-daemon-config\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.192102 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.191269 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5355db2a-b7df-48d3-9554-b5eaa4e937ab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k2vj4\" (UID: \"5355db2a-b7df-48d3-9554-b5eaa4e937ab\") " pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.192102 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.191300 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/da6931b3-d9b0-4c63-a0ba-7eca03b046db-konnectivity-ca\") pod \"konnectivity-agent-jlx72\" (UID: \"da6931b3-d9b0-4c63-a0ba-7eca03b046db\") " pod="kube-system/konnectivity-agent-jlx72" Jul 21 00:16:48.207588 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.207566 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zqwxb" Jul 21 00:16:48.215018 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.214993 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zqwxb" Jul 21 00:16:48.277453 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.277425 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 21 00:16:48.291861 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.291826 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/00221678-4e87-4266-9c77-51182bb87c2b-host-slash\") pod \"iptables-alerter-6pvbf\" (UID: \"00221678-4e87-4266-9c77-51182bb87c2b\") " pod="openshift-network-operator/iptables-alerter-6pvbf" Jul 21 00:16:48.292021 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.291871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/da6931b3-d9b0-4c63-a0ba-7eca03b046db-agent-certs\") pod \"konnectivity-agent-jlx72\" (UID: \"da6931b3-d9b0-4c63-a0ba-7eca03b046db\") " pod="kube-system/konnectivity-agent-jlx72" Jul 21 00:16:48.292021 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.291896 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-systemd-units\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.292021 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.291920 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-run-ovn\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.292021 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.291953 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5b7f\" (UniqueName: \"kubernetes.io/projected/583c4be0-d3e2-4a43-b91c-d6a8903569da-kube-api-access-j5b7f\") pod \"node-resolver-szxbf\" (UID: \"583c4be0-d3e2-4a43-b91c-d6a8903569da\") " pod="openshift-dns/node-resolver-szxbf" Jul 21 00:16:48.292021 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.291986 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37628504-b773-4dad-a955-2a87d40ab71c-multus-daemon-config\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.292021 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292011 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5355db2a-b7df-48d3-9554-b5eaa4e937ab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k2vj4\" (UID: \"5355db2a-b7df-48d3-9554-b5eaa4e937ab\") " pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.292315 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292046 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/da6931b3-d9b0-4c63-a0ba-7eca03b046db-konnectivity-ca\") pod \"konnectivity-agent-jlx72\" (UID: \"da6931b3-d9b0-4c63-a0ba-7eca03b046db\") " pod="kube-system/konnectivity-agent-jlx72" Jul 21 00:16:48.292315 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-multus-cni-dir\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.292315 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292109 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48d1f8d1-615b-4b43-ab84-f2b22e43a382-sys-fs\") pod \"aws-ebs-csi-driver-node-rnjq8\" (UID: \"48d1f8d1-615b-4b43-ab84-f2b22e43a382\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" Jul 21 00:16:48.292315 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292141 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-host-slash\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.292315 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292172 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5355db2a-b7df-48d3-9554-b5eaa4e937ab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k2vj4\" (UID: \"5355db2a-b7df-48d3-9554-b5eaa4e937ab\") " pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.292315 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292196 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-etc-sysctl-d\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.292315 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292219 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-run-systemd\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.292315 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292248 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-host-run-ovn-kubernetes\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.292315 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292277 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xffqs\" (UniqueName: \"kubernetes.io/projected/982c215c-29b3-45ac-8525-03c61f27e931-kube-api-access-xffqs\") pod \"network-metrics-daemon-n8sh6\" (UID: \"982c215c-29b3-45ac-8525-03c61f27e931\") " pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:16:48.292725 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292323 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5355db2a-b7df-48d3-9554-b5eaa4e937ab-cni-binary-copy\") pod \"multus-additional-cni-plugins-k2vj4\" (UID: \"5355db2a-b7df-48d3-9554-b5eaa4e937ab\") " pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.292725 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292367 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zhfh\" (UniqueName: \"kubernetes.io/projected/2bd25715-f83d-42fa-953e-ba07d8b7da50-kube-api-access-2zhfh\") pod \"node-ca-p76tl\" (UID: \"2bd25715-f83d-42fa-953e-ba07d8b7da50\") " pod="openshift-image-registry/node-ca-p76tl" Jul 21 00:16:48.292725 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292392 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sklg6\" (UniqueName: \"kubernetes.io/projected/00221678-4e87-4266-9c77-51182bb87c2b-kube-api-access-sklg6\") pod \"iptables-alerter-6pvbf\" (UID: \"00221678-4e87-4266-9c77-51182bb87c2b\") " pod="openshift-network-operator/iptables-alerter-6pvbf" Jul 21 00:16:48.292725 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292420 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m8vd\" (UniqueName: \"kubernetes.io/projected/48d1f8d1-615b-4b43-ab84-f2b22e43a382-kube-api-access-2m8vd\") pod \"aws-ebs-csi-driver-node-rnjq8\" (UID: \"48d1f8d1-615b-4b43-ab84-f2b22e43a382\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" Jul 21 00:16:48.292725 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292465 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-var-lib-openvswitch\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.292725 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292520 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-hostroot\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.292725 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292553 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5355db2a-b7df-48d3-9554-b5eaa4e937ab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k2vj4\" (UID: \"5355db2a-b7df-48d3-9554-b5eaa4e937ab\") " pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.292725 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292583 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2bd25715-f83d-42fa-953e-ba07d8b7da50-serviceca\") pod \"node-ca-p76tl\" (UID: \"2bd25715-f83d-42fa-953e-ba07d8b7da50\") " pod="openshift-image-registry/node-ca-p76tl" Jul 21 00:16:48.292725 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292621 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/23335158-224d-46e1-b617-10ffa4acb460-original-pull-secret\") pod \"global-pull-secret-syncer-2psfz\" (UID: \"23335158-224d-46e1-b617-10ffa4acb460\") " pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:16:48.292725 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292645 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-run-openvswitch\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.292725 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292662 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-ovnkube-config\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.292725 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292676 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48d1f8d1-615b-4b43-ab84-f2b22e43a382-registration-dir\") pod \"aws-ebs-csi-driver-node-rnjq8\" (UID: \"48d1f8d1-615b-4b43-ab84-f2b22e43a382\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" Jul 21 00:16:48.292725 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292698 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-multus-socket-dir-parent\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.293329 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292734 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-host-var-lib-cni-bin\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.293329 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292751 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-host-run-multus-certs\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.293329 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292792 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-var-lib-kubelet\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.293329 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292825 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-etc-sysctl-conf\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.293329 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292840 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-lib-modules\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.293329 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292935 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-host\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.293329 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.292962 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48d1f8d1-615b-4b43-ab84-f2b22e43a382-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnjq8\" (UID: \"48d1f8d1-615b-4b43-ab84-f2b22e43a382\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" Jul 21 00:16:48.293664 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.293379 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-host-run-netns\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.293664 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.293435 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6c602047-2c64-4c38-ba0d-8042a77f4e24-etc-tuned\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.293664 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.293482 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c602047-2c64-4c38-ba0d-8042a77f4e24-tmp\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.293664 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.293545 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48d1f8d1-615b-4b43-ab84-f2b22e43a382-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnjq8\" (UID: \"48d1f8d1-615b-4b43-ab84-f2b22e43a382\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" Jul 21 00:16:48.293664 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.293580 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-node-log\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.293664 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.293655 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-run\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.293919 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.293708 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/00221678-4e87-4266-9c77-51182bb87c2b-iptables-alerter-script\") pod \"iptables-alerter-6pvbf\" (UID: \"00221678-4e87-4266-9c77-51182bb87c2b\") " pod="openshift-network-operator/iptables-alerter-6pvbf" Jul 21 00:16:48.293919 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.293744 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-host-var-lib-cni-multus\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.293919 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.293820 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-host-var-lib-cni-multus\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.294302 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.294259 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 21 00:16:48.294473 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.294440 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-node-log\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.294885 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.294859 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-run\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.294978 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.294872 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-systemd-units\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.295305 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.295286 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48d1f8d1-615b-4b43-ab84-f2b22e43a382-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rnjq8\" (UID: \"48d1f8d1-615b-4b43-ab84-f2b22e43a382\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" Jul 21 00:16:48.295648 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.295626 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/00221678-4e87-4266-9c77-51182bb87c2b-iptables-alerter-script\") pod \"iptables-alerter-6pvbf\" (UID: \"00221678-4e87-4266-9c77-51182bb87c2b\") " pod="openshift-network-operator/iptables-alerter-6pvbf" Jul 21 00:16:48.295767 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.295749 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-run-ovn\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.295832 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.295804 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-ovnkube-config\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.296234 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.296215 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-var-lib-openvswitch\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.296326 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.296281 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-hostroot\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.296458 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.296442 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5355db2a-b7df-48d3-9554-b5eaa4e937ab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k2vj4\" (UID: \"5355db2a-b7df-48d3-9554-b5eaa4e937ab\") " pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.296640 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.296624 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-run-openvswitch\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.296757 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.296733 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/37628504-b773-4dad-a955-2a87d40ab71c-multus-daemon-config\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.296881 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.296863 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-etc-sysctl-d\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.296959 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.296944 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-run-systemd\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.297017 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.297006 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-host-run-ovn-kubernetes\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.297213 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.297190 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5355db2a-b7df-48d3-9554-b5eaa4e937ab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k2vj4\" (UID: \"5355db2a-b7df-48d3-9554-b5eaa4e937ab\") " pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.297652 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.297617 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5355db2a-b7df-48d3-9554-b5eaa4e937ab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k2vj4\" (UID: \"5355db2a-b7df-48d3-9554-b5eaa4e937ab\") " pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.297857 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.297837 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5355db2a-b7df-48d3-9554-b5eaa4e937ab-cni-binary-copy\") pod \"multus-additional-cni-plugins-k2vj4\" (UID: \"5355db2a-b7df-48d3-9554-b5eaa4e937ab\") " pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.297928 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.297889 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/00221678-4e87-4266-9c77-51182bb87c2b-host-slash\") pod \"iptables-alerter-6pvbf\" (UID: \"00221678-4e87-4266-9c77-51182bb87c2b\") " pod="openshift-network-operator/iptables-alerter-6pvbf" Jul 21 00:16:48.297981 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.297932 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-etc-kubernetes\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.297981 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.297963 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-etc-openvswitch\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.298094 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.297991 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs\") pod \"network-metrics-daemon-n8sh6\" (UID: \"982c215c-29b3-45ac-8525-03c61f27e931\") " pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:16:48.298094 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.298038 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7lfv\" (UniqueName: \"kubernetes.io/projected/6c602047-2c64-4c38-ba0d-8042a77f4e24-kube-api-access-l7lfv\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.298094 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.298066 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37628504-b773-4dad-a955-2a87d40ab71c-cni-binary-copy\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.298278 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.298111 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-host-var-lib-kubelet\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.298278 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.298174 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-env-overrides\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.298278 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.298163 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/da6931b3-d9b0-4c63-a0ba-7eca03b046db-konnectivity-ca\") pod \"konnectivity-agent-jlx72\" (UID: \"da6931b3-d9b0-4c63-a0ba-7eca03b046db\") " pod="kube-system/konnectivity-agent-jlx72" Jul 21 00:16:48.298278 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.298246 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c602047-2c64-4c38-ba0d-8042a77f4e24-tmp\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.298278 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.298266 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-host-slash\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.298510 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:48.298314 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:16:48.298510 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.298358 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-var-lib-kubelet\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.298510 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:48.298434 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs podName:982c215c-29b3-45ac-8525-03c61f27e931 nodeName:}" failed. No retries permitted until 2026-07-21 00:16:48.798399322 +0000 UTC m=+3.013779643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs") pod "network-metrics-daemon-n8sh6" (UID: "982c215c-29b3-45ac-8525-03c61f27e931") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:16:48.298510 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.298440 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48d1f8d1-615b-4b43-ab84-f2b22e43a382-registration-dir\") pod \"aws-ebs-csi-driver-node-rnjq8\" (UID: \"48d1f8d1-615b-4b43-ab84-f2b22e43a382\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" Jul 21 00:16:48.298510 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.298490 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6c602047-2c64-4c38-ba0d-8042a77f4e24-etc-tuned\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.298773 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.298512 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-multus-socket-dir-parent\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.298773 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.298579 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-host-run-multus-certs\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.298773 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.298673 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-lib-modules\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.298773 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.298731 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-host-var-lib-cni-bin\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.298773 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.298772 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-ovnkube-script-lib\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.299017 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.298808 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5355db2a-b7df-48d3-9554-b5eaa4e937ab-system-cni-dir\") pod \"multus-additional-cni-plugins-k2vj4\" (UID: \"5355db2a-b7df-48d3-9554-b5eaa4e937ab\") " pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.299017 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.298883 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-etc-openvswitch\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.299017 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.298937 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-etc-kubernetes\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.299017 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.298970 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/23335158-224d-46e1-b617-10ffa4acb460-dbus\") pod \"global-pull-secret-syncer-2psfz\" (UID: \"23335158-224d-46e1-b617-10ffa4acb460\") " pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:16:48.299017 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.299003 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37628504-b773-4dad-a955-2a87d40ab71c-cni-binary-copy\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.299246 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.299005 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-host-cni-netd\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.299246 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.299079 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-etc-sysctl-conf\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.299246 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.299090 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48d1f8d1-615b-4b43-ab84-f2b22e43a382-sys-fs\") pod \"aws-ebs-csi-driver-node-rnjq8\" (UID: \"48d1f8d1-615b-4b43-ab84-f2b22e43a382\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" Jul 21 00:16:48.299246 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.299088 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-etc-kubernetes\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.299246 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.299165 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-host-cni-netd\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.299246 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.299175 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-host-var-lib-kubelet\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.299246 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.299186 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-etc-kubernetes\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.299246 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.299226 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5355db2a-b7df-48d3-9554-b5eaa4e937ab-system-cni-dir\") pod \"multus-additional-cni-plugins-k2vj4\" (UID: \"5355db2a-b7df-48d3-9554-b5eaa4e937ab\") " pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.299614 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.299292 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-host\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.299614 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.299283 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-multus-cni-dir\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.299614 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.299446 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48d1f8d1-615b-4b43-ab84-f2b22e43a382-etc-selinux\") pod \"aws-ebs-csi-driver-node-rnjq8\" (UID: \"48d1f8d1-615b-4b43-ab84-f2b22e43a382\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" Jul 21 00:16:48.299816 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.299796 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-env-overrides\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.299892 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.299853 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.299941 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.299916 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r76wr\" (UniqueName: \"kubernetes.io/projected/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-kube-api-access-r76wr\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.299990 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.299959 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/583c4be0-d3e2-4a43-b91c-d6a8903569da-hosts-file\") pod \"node-resolver-szxbf\" (UID: \"583c4be0-d3e2-4a43-b91c-d6a8903569da\") " pod="openshift-dns/node-resolver-szxbf" Jul 21 00:16:48.300042 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.299994 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5355db2a-b7df-48d3-9554-b5eaa4e937ab-os-release\") pod \"multus-additional-cni-plugins-k2vj4\" (UID: \"5355db2a-b7df-48d3-9554-b5eaa4e937ab\") " pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.300042 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.300029 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-etc-systemd\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.300127 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.300061 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-sys\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.300332 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.300090 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-host-run-netns\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.300494 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.300372 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-host-run-k8s-cni-cncf-io\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.300494 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.300421 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/583c4be0-d3e2-4a43-b91c-d6a8903569da-hosts-file\") pod \"node-resolver-szxbf\" (UID: \"583c4be0-d3e2-4a43-b91c-d6a8903569da\") " pod="openshift-dns/node-resolver-szxbf" Jul 21 00:16:48.300762 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.300727 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-ovnkube-script-lib\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.300873 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.300714 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-etc-systemd\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.300873 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.300792 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-sys\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.300873 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.300827 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-host-kubelet\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.300873 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.300844 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.301079 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.300643 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5355db2a-b7df-48d3-9554-b5eaa4e937ab-os-release\") pod \"multus-additional-cni-plugins-k2vj4\" (UID: \"5355db2a-b7df-48d3-9554-b5eaa4e937ab\") " pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.301079 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.300870 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-ovn-node-metrics-cert\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.301079 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.300903 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-host-kubelet\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.301079 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.300966 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-etc-sysconfig\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.301079 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.300994 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-host-run-k8s-cni-cncf-io\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.301079 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.300999 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583c4be0-d3e2-4a43-b91c-d6a8903569da-tmp-dir\") pod \"node-resolver-szxbf\" (UID: \"583c4be0-d3e2-4a43-b91c-d6a8903569da\") " pod="openshift-dns/node-resolver-szxbf" Jul 21 00:16:48.301079 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.301045 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-etc-sysconfig\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.301079 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.301050 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48d1f8d1-615b-4b43-ab84-f2b22e43a382-socket-dir\") pod \"aws-ebs-csi-driver-node-rnjq8\" (UID: \"48d1f8d1-615b-4b43-ab84-f2b22e43a382\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" Jul 21 00:16:48.301457 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.301163 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48d1f8d1-615b-4b43-ab84-f2b22e43a382-socket-dir\") pod \"aws-ebs-csi-driver-node-rnjq8\" (UID: \"48d1f8d1-615b-4b43-ab84-f2b22e43a382\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" Jul 21 00:16:48.301457 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.301232 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-multus-conf-dir\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.301457 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.301272 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48d1f8d1-615b-4b43-ab84-f2b22e43a382-device-dir\") pod \"aws-ebs-csi-driver-node-rnjq8\" (UID: \"48d1f8d1-615b-4b43-ab84-f2b22e43a382\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" Jul 21 00:16:48.301457 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.301334 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583c4be0-d3e2-4a43-b91c-d6a8903569da-tmp-dir\") pod \"node-resolver-szxbf\" (UID: \"583c4be0-d3e2-4a43-b91c-d6a8903569da\") " pod="openshift-dns/node-resolver-szxbf" Jul 21 00:16:48.301926 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.301882 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-multus-conf-dir\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.302017 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.301916 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48d1f8d1-615b-4b43-ab84-f2b22e43a382-device-dir\") pod \"aws-ebs-csi-driver-node-rnjq8\" (UID: \"48d1f8d1-615b-4b43-ab84-f2b22e43a382\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" Jul 21 00:16:48.302229 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.302208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-etc-modprobe-d\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.302300 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.302256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2bd25715-f83d-42fa-953e-ba07d8b7da50-host\") pod \"node-ca-p76tl\" (UID: \"2bd25715-f83d-42fa-953e-ba07d8b7da50\") " pod="openshift-image-registry/node-ca-p76tl" Jul 21 00:16:48.302627 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.302328 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/23335158-224d-46e1-b617-10ffa4acb460-kubelet-config\") pod \"global-pull-secret-syncer-2psfz\" (UID: \"23335158-224d-46e1-b617-10ffa4acb460\") " pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:16:48.302627 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.302413 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6c602047-2c64-4c38-ba0d-8042a77f4e24-etc-modprobe-d\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.302627 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.302423 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-system-cni-dir\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.302627 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.302469 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-cnibin\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.302627 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.302549 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-cnibin\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.302936 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.302592 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhcws\" (UniqueName: \"kubernetes.io/projected/37628504-b773-4dad-a955-2a87d40ab71c-kube-api-access-dhcws\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.302936 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.302594 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-system-cni-dir\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.303934 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.303194 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-log-socket\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.303934 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.303267 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-host-cni-bin\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.303934 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.303307 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5355db2a-b7df-48d3-9554-b5eaa4e937ab-cnibin\") pod \"multus-additional-cni-plugins-k2vj4\" (UID: \"5355db2a-b7df-48d3-9554-b5eaa4e937ab\") " pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.303934 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.303341 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-log-socket\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.303934 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.303346 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zp89\" (UniqueName: \"kubernetes.io/projected/5355db2a-b7df-48d3-9554-b5eaa4e937ab-kube-api-access-8zp89\") pod \"multus-additional-cni-plugins-k2vj4\" (UID: \"5355db2a-b7df-48d3-9554-b5eaa4e937ab\") " pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.303934 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.303374 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-host-cni-bin\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.303934 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.303401 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/da6931b3-d9b0-4c63-a0ba-7eca03b046db-agent-certs\") pod \"konnectivity-agent-jlx72\" (UID: \"da6931b3-d9b0-4c63-a0ba-7eca03b046db\") " pod="kube-system/konnectivity-agent-jlx72" Jul 21 00:16:48.303934 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.303438 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnbcl\" (UniqueName: \"kubernetes.io/projected/e6e90e27-ad89-49df-904c-d28dd5ecaea4-kube-api-access-tnbcl\") pod \"network-check-target-98dgh\" (UID: \"e6e90e27-ad89-49df-904c-d28dd5ecaea4\") " pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:16:48.303934 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.303563 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5355db2a-b7df-48d3-9554-b5eaa4e937ab-cnibin\") pod \"multus-additional-cni-plugins-k2vj4\" (UID: \"5355db2a-b7df-48d3-9554-b5eaa4e937ab\") " pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.303934 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.303562 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-ovn-node-metrics-cert\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.303934 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.303653 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-os-release\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.303934 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.303698 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-host-run-netns\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.303934 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.303810 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-os-release\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.303934 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.303856 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37628504-b773-4dad-a955-2a87d40ab71c-host-run-netns\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.305086 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.304700 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-117.ec2.internal" event={"ID":"30cbc68e3783a788ccf4c7e2f8cc9298","Type":"ContainerStarted","Data":"f9daae0f7b9e3f5ded1b91e0aec50e55a21ec9d9274d78ee8eaae329be8e44cd"} Jul 21 00:16:48.305885 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.305857 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-117.ec2.internal" event={"ID":"0b00f410120824644017f2fc5f606062","Type":"ContainerStarted","Data":"16ee0c7dc74a9695c9b1551b0d332a46062c8f59b4a0e2e1b54cf729ccd3a56d"} Jul 21 00:16:48.310708 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.310195 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m8vd\" (UniqueName: \"kubernetes.io/projected/48d1f8d1-615b-4b43-ab84-f2b22e43a382-kube-api-access-2m8vd\") pod \"aws-ebs-csi-driver-node-rnjq8\" (UID: \"48d1f8d1-615b-4b43-ab84-f2b22e43a382\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" Jul 21 00:16:48.310708 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.310658 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xffqs\" (UniqueName: \"kubernetes.io/projected/982c215c-29b3-45ac-8525-03c61f27e931-kube-api-access-xffqs\") pod \"network-metrics-daemon-n8sh6\" (UID: \"982c215c-29b3-45ac-8525-03c61f27e931\") " pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:16:48.310949 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.310923 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5b7f\" (UniqueName: \"kubernetes.io/projected/583c4be0-d3e2-4a43-b91c-d6a8903569da-kube-api-access-j5b7f\") pod \"node-resolver-szxbf\" (UID: \"583c4be0-d3e2-4a43-b91c-d6a8903569da\") " pod="openshift-dns/node-resolver-szxbf" Jul 21 00:16:48.311063 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.310995 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sklg6\" (UniqueName: \"kubernetes.io/projected/00221678-4e87-4266-9c77-51182bb87c2b-kube-api-access-sklg6\") pod \"iptables-alerter-6pvbf\" (UID: \"00221678-4e87-4266-9c77-51182bb87c2b\") " pod="openshift-network-operator/iptables-alerter-6pvbf" Jul 21 00:16:48.311765 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.311743 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r76wr\" (UniqueName: \"kubernetes.io/projected/af62f4c5-4beb-44e6-903e-5c71ede9a3cd-kube-api-access-r76wr\") pod \"ovnkube-node-mzgzs\" (UID: \"af62f4c5-4beb-44e6-903e-5c71ede9a3cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.312906 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.312884 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhcws\" (UniqueName: \"kubernetes.io/projected/37628504-b773-4dad-a955-2a87d40ab71c-kube-api-access-dhcws\") pod \"multus-scmjq\" (UID: \"37628504-b773-4dad-a955-2a87d40ab71c\") " pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.313622 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:48.313354 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 00:16:48.313622 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:48.313377 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 00:16:48.313622 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:48.313392 2580 projected.go:196] Error preparing data for projected volume kube-api-access-tnbcl for pod openshift-network-diagnostics/network-check-target-98dgh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:16:48.313622 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:48.313473 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6e90e27-ad89-49df-904c-d28dd5ecaea4-kube-api-access-tnbcl podName:e6e90e27-ad89-49df-904c-d28dd5ecaea4 nodeName:}" failed. No retries permitted until 2026-07-21 00:16:48.813455723 +0000 UTC m=+3.028836026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tnbcl" (UniqueName: "kubernetes.io/projected/e6e90e27-ad89-49df-904c-d28dd5ecaea4-kube-api-access-tnbcl") pod "network-check-target-98dgh" (UID: "e6e90e27-ad89-49df-904c-d28dd5ecaea4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:16:48.313622 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.313545 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7lfv\" (UniqueName: \"kubernetes.io/projected/6c602047-2c64-4c38-ba0d-8042a77f4e24-kube-api-access-l7lfv\") pod \"tuned-fcl75\" (UID: \"6c602047-2c64-4c38-ba0d-8042a77f4e24\") " pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.316561 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.316535 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zp89\" (UniqueName: \"kubernetes.io/projected/5355db2a-b7df-48d3-9554-b5eaa4e937ab-kube-api-access-8zp89\") pod \"multus-additional-cni-plugins-k2vj4\" (UID: \"5355db2a-b7df-48d3-9554-b5eaa4e937ab\") " pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.405056 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.404958 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2bd25715-f83d-42fa-953e-ba07d8b7da50-host\") pod \"node-ca-p76tl\" (UID: \"2bd25715-f83d-42fa-953e-ba07d8b7da50\") " pod="openshift-image-registry/node-ca-p76tl" Jul 21 00:16:48.405056 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.405006 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/23335158-224d-46e1-b617-10ffa4acb460-kubelet-config\") pod \"global-pull-secret-syncer-2psfz\" (UID: \"23335158-224d-46e1-b617-10ffa4acb460\") " pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:16:48.405261 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.405064 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zhfh\" (UniqueName: \"kubernetes.io/projected/2bd25715-f83d-42fa-953e-ba07d8b7da50-kube-api-access-2zhfh\") pod \"node-ca-p76tl\" (UID: \"2bd25715-f83d-42fa-953e-ba07d8b7da50\") " pod="openshift-image-registry/node-ca-p76tl" Jul 21 00:16:48.405261 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.405102 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2bd25715-f83d-42fa-953e-ba07d8b7da50-serviceca\") pod \"node-ca-p76tl\" (UID: \"2bd25715-f83d-42fa-953e-ba07d8b7da50\") " pod="openshift-image-registry/node-ca-p76tl" Jul 21 00:16:48.405261 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.405125 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/23335158-224d-46e1-b617-10ffa4acb460-original-pull-secret\") pod \"global-pull-secret-syncer-2psfz\" (UID: \"23335158-224d-46e1-b617-10ffa4acb460\") " pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:16:48.405261 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.405096 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2bd25715-f83d-42fa-953e-ba07d8b7da50-host\") pod \"node-ca-p76tl\" (UID: \"2bd25715-f83d-42fa-953e-ba07d8b7da50\") " pod="openshift-image-registry/node-ca-p76tl" Jul 21 00:16:48.405261 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.405199 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/23335158-224d-46e1-b617-10ffa4acb460-kubelet-config\") pod \"global-pull-secret-syncer-2psfz\" (UID: \"23335158-224d-46e1-b617-10ffa4acb460\") " pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:16:48.405261 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:48.405249 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 00:16:48.405457 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.405264 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/23335158-224d-46e1-b617-10ffa4acb460-dbus\") pod \"global-pull-secret-syncer-2psfz\" (UID: \"23335158-224d-46e1-b617-10ffa4acb460\") " pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:16:48.405457 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:48.405308 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23335158-224d-46e1-b617-10ffa4acb460-original-pull-secret podName:23335158-224d-46e1-b617-10ffa4acb460 nodeName:}" failed. No retries permitted until 2026-07-21 00:16:48.905288637 +0000 UTC m=+3.120668961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/23335158-224d-46e1-b617-10ffa4acb460-original-pull-secret") pod "global-pull-secret-syncer-2psfz" (UID: "23335158-224d-46e1-b617-10ffa4acb460") : object "kube-system"/"original-pull-secret" not registered Jul 21 00:16:48.405457 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.405331 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/23335158-224d-46e1-b617-10ffa4acb460-dbus\") pod \"global-pull-secret-syncer-2psfz\" (UID: \"23335158-224d-46e1-b617-10ffa4acb460\") " pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:16:48.405584 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.405567 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2bd25715-f83d-42fa-953e-ba07d8b7da50-serviceca\") pod \"node-ca-p76tl\" (UID: \"2bd25715-f83d-42fa-953e-ba07d8b7da50\") " pod="openshift-image-registry/node-ca-p76tl" Jul 21 00:16:48.413463 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.413431 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 00:16:48.417309 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.417285 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zhfh\" (UniqueName: \"kubernetes.io/projected/2bd25715-f83d-42fa-953e-ba07d8b7da50-kube-api-access-2zhfh\") pod \"node-ca-p76tl\" (UID: \"2bd25715-f83d-42fa-953e-ba07d8b7da50\") " pod="openshift-image-registry/node-ca-p76tl" Jul 21 00:16:48.475283 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.475248 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:16:48.484129 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.484096 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-szxbf" Jul 21 00:16:48.487497 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.487479 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 00:16:48.496617 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.496577 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k2vj4" Jul 21 00:16:48.503548 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.503525 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-scmjq" Jul 21 00:16:48.509523 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.509500 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6pvbf" Jul 21 00:16:48.517476 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.517453 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jlx72" Jul 21 00:16:48.524416 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.524392 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fcl75" Jul 21 00:16:48.531393 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.531375 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" Jul 21 00:16:48.538177 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.538156 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p76tl" Jul 21 00:16:48.808366 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.808295 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs\") pod \"network-metrics-daemon-n8sh6\" (UID: \"982c215c-29b3-45ac-8525-03c61f27e931\") " pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:16:48.808500 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:48.808429 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:16:48.808500 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:48.808491 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs podName:982c215c-29b3-45ac-8525-03c61f27e931 nodeName:}" failed. No retries permitted until 2026-07-21 00:16:49.80846992 +0000 UTC m=+4.023850223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs") pod "network-metrics-daemon-n8sh6" (UID: "982c215c-29b3-45ac-8525-03c61f27e931") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:16:48.908812 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.908779 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnbcl\" (UniqueName: \"kubernetes.io/projected/e6e90e27-ad89-49df-904c-d28dd5ecaea4-kube-api-access-tnbcl\") pod \"network-check-target-98dgh\" (UID: \"e6e90e27-ad89-49df-904c-d28dd5ecaea4\") " pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:16:48.908812 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:48.908820 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/23335158-224d-46e1-b617-10ffa4acb460-original-pull-secret\") pod \"global-pull-secret-syncer-2psfz\" (UID: \"23335158-224d-46e1-b617-10ffa4acb460\") " pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:16:48.909006 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:48.908942 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 00:16:48.909006 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:48.908950 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 00:16:48.909006 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:48.908963 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 00:16:48.909006 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:48.908972 2580 projected.go:196] Error preparing data for projected volume kube-api-access-tnbcl for pod openshift-network-diagnostics/network-check-target-98dgh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:16:48.909125 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:48.909012 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23335158-224d-46e1-b617-10ffa4acb460-original-pull-secret podName:23335158-224d-46e1-b617-10ffa4acb460 nodeName:}" failed. No retries permitted until 2026-07-21 00:16:49.90899457 +0000 UTC m=+4.124374878 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/23335158-224d-46e1-b617-10ffa4acb460-original-pull-secret") pod "global-pull-secret-syncer-2psfz" (UID: "23335158-224d-46e1-b617-10ffa4acb460") : object "kube-system"/"original-pull-secret" not registered Jul 21 00:16:48.909125 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:48.909025 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6e90e27-ad89-49df-904c-d28dd5ecaea4-kube-api-access-tnbcl podName:e6e90e27-ad89-49df-904c-d28dd5ecaea4 nodeName:}" failed. No retries permitted until 2026-07-21 00:16:49.909019479 +0000 UTC m=+4.124399782 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tnbcl" (UniqueName: "kubernetes.io/projected/e6e90e27-ad89-49df-904c-d28dd5ecaea4-kube-api-access-tnbcl") pod "network-check-target-98dgh" (UID: "e6e90e27-ad89-49df-904c-d28dd5ecaea4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:16:49.178337 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:49.178309 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c602047_2c64_4c38_ba0d_8042a77f4e24.slice/crio-c053e7f5e1d28dc10460500322c849ac62fa49d46f4d198fffc68e79dfb5e3a7 WatchSource:0}: Error finding container c053e7f5e1d28dc10460500322c849ac62fa49d46f4d198fffc68e79dfb5e3a7: Status 404 returned error can't find the container with id c053e7f5e1d28dc10460500322c849ac62fa49d46f4d198fffc68e79dfb5e3a7 Jul 21 00:16:49.186959 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:49.186922 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37628504_b773_4dad_a955_2a87d40ab71c.slice/crio-c3619f7a8e9e9ef3207e97a4c1e3eec1b0c999ae2c8001ffe713cde5fd94e204 WatchSource:0}: Error finding container c3619f7a8e9e9ef3207e97a4c1e3eec1b0c999ae2c8001ffe713cde5fd94e204: Status 404 returned error can't find the container with id c3619f7a8e9e9ef3207e97a4c1e3eec1b0c999ae2c8001ffe713cde5fd94e204 Jul 21 00:16:49.194956 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:49.194927 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd25715_f83d_42fa_953e_ba07d8b7da50.slice/crio-39f45fd5c152ea14a5c77292cc15179ce6c9f1720419a8f27e5ac53e5dee9f86 WatchSource:0}: Error finding container 39f45fd5c152ea14a5c77292cc15179ce6c9f1720419a8f27e5ac53e5dee9f86: Status 404 returned error can't find the container with id 39f45fd5c152ea14a5c77292cc15179ce6c9f1720419a8f27e5ac53e5dee9f86 Jul 21 00:16:49.195642 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:49.195557 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d1f8d1_615b_4b43_ab84_f2b22e43a382.slice/crio-ace27b4b02d9ab3cd8e37ebf237071cd14ebedda5384c4bf1b1ffd2f61c50d21 WatchSource:0}: Error finding container ace27b4b02d9ab3cd8e37ebf237071cd14ebedda5384c4bf1b1ffd2f61c50d21: Status 404 returned error can't find the container with id ace27b4b02d9ab3cd8e37ebf237071cd14ebedda5384c4bf1b1ffd2f61c50d21 Jul 21 00:16:49.201983 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:49.201930 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5355db2a_b7df_48d3_9554_b5eaa4e937ab.slice/crio-5654d4a5463f4cc5fd2dd4feb11078ed3cb8517019bde7919db79f0d71ab4536 WatchSource:0}: Error finding container 5654d4a5463f4cc5fd2dd4feb11078ed3cb8517019bde7919db79f0d71ab4536: Status 404 returned error can't find the container with id 5654d4a5463f4cc5fd2dd4feb11078ed3cb8517019bde7919db79f0d71ab4536 Jul 21 00:16:49.203910 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:16:49.203868 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583c4be0_d3e2_4a43_b91c_d6a8903569da.slice/crio-afd3a88fb777b05ebf7b8fe1c6075a691b563ac35cfce4544b0fa898b3f92e03 WatchSource:0}: Error finding container afd3a88fb777b05ebf7b8fe1c6075a691b563ac35cfce4544b0fa898b3f92e03: Status 404 returned error can't find the container with id afd3a88fb777b05ebf7b8fe1c6075a691b563ac35cfce4544b0fa898b3f92e03 Jul 21 00:16:49.217249 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:49.217223 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 00:11:48 +0000 UTC" deadline="2027-12-16 16:40:14.050680712 +0000 UTC" Jul 21 00:16:49.217249 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:49.217247 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12328h23m24.833437006s" Jul 21 00:16:49.290973 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:49.290791 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:16:49.291051 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:49.290799 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:16:49.291083 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:49.291063 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2psfz" podUID="23335158-224d-46e1-b617-10ffa4acb460" Jul 21 00:16:49.291139 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:49.291122 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8sh6" podUID="982c215c-29b3-45ac-8525-03c61f27e931" Jul 21 00:16:49.319622 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:49.309212 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k2vj4" event={"ID":"5355db2a-b7df-48d3-9554-b5eaa4e937ab","Type":"ContainerStarted","Data":"5654d4a5463f4cc5fd2dd4feb11078ed3cb8517019bde7919db79f0d71ab4536"} Jul 21 00:16:49.319622 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:49.310429 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6pvbf" event={"ID":"00221678-4e87-4266-9c77-51182bb87c2b","Type":"ContainerStarted","Data":"20daed0147069e914d6e61eb84dc48bd0089bc01bbb04c2af8d5d58dad2e6a26"} Jul 21 00:16:49.319622 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:49.311453 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p76tl" event={"ID":"2bd25715-f83d-42fa-953e-ba07d8b7da50","Type":"ContainerStarted","Data":"39f45fd5c152ea14a5c77292cc15179ce6c9f1720419a8f27e5ac53e5dee9f86"} Jul 21 00:16:49.319622 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:49.312631 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jlx72" event={"ID":"da6931b3-d9b0-4c63-a0ba-7eca03b046db","Type":"ContainerStarted","Data":"f057e2a3a6b364fb633d5106d278511e44302404914d159d809b12c6fe7e5d0b"} Jul 21 00:16:49.319622 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:49.313844 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" event={"ID":"af62f4c5-4beb-44e6-903e-5c71ede9a3cd","Type":"ContainerStarted","Data":"c6e57bdbc3bfde43f25172680ac0ea2bad05dcce180112b21d72919590d714f8"} Jul 21 00:16:49.319622 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:49.314958 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-scmjq" event={"ID":"37628504-b773-4dad-a955-2a87d40ab71c","Type":"ContainerStarted","Data":"c3619f7a8e9e9ef3207e97a4c1e3eec1b0c999ae2c8001ffe713cde5fd94e204"} Jul 21 00:16:49.319622 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:49.315925 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-szxbf" event={"ID":"583c4be0-d3e2-4a43-b91c-d6a8903569da","Type":"ContainerStarted","Data":"afd3a88fb777b05ebf7b8fe1c6075a691b563ac35cfce4544b0fa898b3f92e03"} Jul 21 00:16:49.319622 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:49.317042 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" event={"ID":"48d1f8d1-615b-4b43-ab84-f2b22e43a382","Type":"ContainerStarted","Data":"ace27b4b02d9ab3cd8e37ebf237071cd14ebedda5384c4bf1b1ffd2f61c50d21"} Jul 21 00:16:49.319622 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:49.318146 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fcl75" event={"ID":"6c602047-2c64-4c38-ba0d-8042a77f4e24","Type":"ContainerStarted","Data":"c053e7f5e1d28dc10460500322c849ac62fa49d46f4d198fffc68e79dfb5e3a7"} Jul 21 00:16:49.320069 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:49.320024 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-117.ec2.internal" event={"ID":"30cbc68e3783a788ccf4c7e2f8cc9298","Type":"ContainerStarted","Data":"79869c12d22db4979f323e62dabae779d9a301bf1ba1e93286b49a69fe92e188"} Jul 21 00:16:49.335965 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:49.335903 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-117.ec2.internal" podStartSLOduration=2.335883764 podStartE2EDuration="2.335883764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 00:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:16:49.33546437 +0000 UTC m=+3.550844696" watchObservedRunningTime="2026-07-21 00:16:49.335883764 +0000 UTC m=+3.551264092" Jul 21 00:16:49.815736 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:49.815080 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs\") pod \"network-metrics-daemon-n8sh6\" (UID: \"982c215c-29b3-45ac-8525-03c61f27e931\") " pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:16:49.815736 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:49.815254 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:16:49.815736 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:49.815316 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs podName:982c215c-29b3-45ac-8525-03c61f27e931 nodeName:}" failed. No retries permitted until 2026-07-21 00:16:51.815297466 +0000 UTC m=+6.030677775 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs") pod "network-metrics-daemon-n8sh6" (UID: "982c215c-29b3-45ac-8525-03c61f27e931") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:16:49.915577 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:49.915542 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnbcl\" (UniqueName: \"kubernetes.io/projected/e6e90e27-ad89-49df-904c-d28dd5ecaea4-kube-api-access-tnbcl\") pod \"network-check-target-98dgh\" (UID: \"e6e90e27-ad89-49df-904c-d28dd5ecaea4\") " pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:16:49.915759 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:49.915617 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/23335158-224d-46e1-b617-10ffa4acb460-original-pull-secret\") pod \"global-pull-secret-syncer-2psfz\" (UID: \"23335158-224d-46e1-b617-10ffa4acb460\") " pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:16:49.915833 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:49.915816 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 00:16:49.915921 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:49.915908 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23335158-224d-46e1-b617-10ffa4acb460-original-pull-secret podName:23335158-224d-46e1-b617-10ffa4acb460 nodeName:}" failed. No retries permitted until 2026-07-21 00:16:51.9158898 +0000 UTC m=+6.131270122 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/23335158-224d-46e1-b617-10ffa4acb460-original-pull-secret") pod "global-pull-secret-syncer-2psfz" (UID: "23335158-224d-46e1-b617-10ffa4acb460") : object "kube-system"/"original-pull-secret" not registered Jul 21 00:16:49.916018 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:49.916005 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 00:16:49.916072 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:49.916024 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 00:16:49.916072 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:49.916036 2580 projected.go:196] Error preparing data for projected volume kube-api-access-tnbcl for pod openshift-network-diagnostics/network-check-target-98dgh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:16:49.916178 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:49.916073 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6e90e27-ad89-49df-904c-d28dd5ecaea4-kube-api-access-tnbcl podName:e6e90e27-ad89-49df-904c-d28dd5ecaea4 nodeName:}" failed. No retries permitted until 2026-07-21 00:16:51.916060998 +0000 UTC m=+6.131441316 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tnbcl" (UniqueName: "kubernetes.io/projected/e6e90e27-ad89-49df-904c-d28dd5ecaea4-kube-api-access-tnbcl") pod "network-check-target-98dgh" (UID: "e6e90e27-ad89-49df-904c-d28dd5ecaea4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:16:50.218932 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:50.218821 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 00:11:48 +0000 UTC" deadline="2028-04-27 10:10:09.438286164 +0000 UTC" Jul 21 00:16:50.218932 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:50.218882 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15513h53m19.219408597s" Jul 21 00:16:50.293441 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:50.293410 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:16:50.293640 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:50.293537 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98dgh" podUID="e6e90e27-ad89-49df-904c-d28dd5ecaea4" Jul 21 00:16:50.333005 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:50.332958 2580 generic.go:356] "Generic (PLEG): container finished" podID="0b00f410120824644017f2fc5f606062" containerID="f000a1b872f7b76a37a934b22e6461fa7932af82f84409c42e5b0e00466fb66f" exitCode=0 Jul 21 00:16:50.333882 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:50.333852 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-117.ec2.internal" event={"ID":"0b00f410120824644017f2fc5f606062","Type":"ContainerDied","Data":"f000a1b872f7b76a37a934b22e6461fa7932af82f84409c42e5b0e00466fb66f"} Jul 21 00:16:51.291096 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:51.291067 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:16:51.291537 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:51.291197 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8sh6" podUID="982c215c-29b3-45ac-8525-03c61f27e931" Jul 21 00:16:51.291640 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:51.291591 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:16:51.291749 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:51.291727 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2psfz" podUID="23335158-224d-46e1-b617-10ffa4acb460" Jul 21 00:16:51.349915 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:51.349227 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-117.ec2.internal" event={"ID":"0b00f410120824644017f2fc5f606062","Type":"ContainerStarted","Data":"02a66d3a9febe06c45aa39e52137034d740213e975682daca59d754f5a8fb309"} Jul 21 00:16:51.834023 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:51.833433 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs\") pod \"network-metrics-daemon-n8sh6\" (UID: \"982c215c-29b3-45ac-8525-03c61f27e931\") " pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:16:51.834023 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:51.833622 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:16:51.834023 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:51.833689 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs podName:982c215c-29b3-45ac-8525-03c61f27e931 nodeName:}" failed. No retries permitted until 2026-07-21 00:16:55.833667305 +0000 UTC m=+10.049047627 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs") pod "network-metrics-daemon-n8sh6" (UID: "982c215c-29b3-45ac-8525-03c61f27e931") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:16:51.934713 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:51.933787 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/23335158-224d-46e1-b617-10ffa4acb460-original-pull-secret\") pod \"global-pull-secret-syncer-2psfz\" (UID: \"23335158-224d-46e1-b617-10ffa4acb460\") " pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:16:51.934713 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:51.933910 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnbcl\" (UniqueName: \"kubernetes.io/projected/e6e90e27-ad89-49df-904c-d28dd5ecaea4-kube-api-access-tnbcl\") pod \"network-check-target-98dgh\" (UID: \"e6e90e27-ad89-49df-904c-d28dd5ecaea4\") " pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:16:51.934713 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:51.934060 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 00:16:51.934713 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:51.934077 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 00:16:51.934713 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:51.934168 2580 projected.go:196] Error preparing data for projected volume kube-api-access-tnbcl for pod openshift-network-diagnostics/network-check-target-98dgh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:16:51.934713 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:51.934227 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6e90e27-ad89-49df-904c-d28dd5ecaea4-kube-api-access-tnbcl podName:e6e90e27-ad89-49df-904c-d28dd5ecaea4 nodeName:}" failed. No retries permitted until 2026-07-21 00:16:55.934208471 +0000 UTC m=+10.149588779 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tnbcl" (UniqueName: "kubernetes.io/projected/e6e90e27-ad89-49df-904c-d28dd5ecaea4-kube-api-access-tnbcl") pod "network-check-target-98dgh" (UID: "e6e90e27-ad89-49df-904c-d28dd5ecaea4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:16:51.934713 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:51.934614 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 00:16:51.934713 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:51.934668 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23335158-224d-46e1-b617-10ffa4acb460-original-pull-secret podName:23335158-224d-46e1-b617-10ffa4acb460 nodeName:}" failed. No retries permitted until 2026-07-21 00:16:55.934652606 +0000 UTC m=+10.150032928 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/23335158-224d-46e1-b617-10ffa4acb460-original-pull-secret") pod "global-pull-secret-syncer-2psfz" (UID: "23335158-224d-46e1-b617-10ffa4acb460") : object "kube-system"/"original-pull-secret" not registered Jul 21 00:16:52.292342 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:52.292251 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:16:52.292780 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:52.292408 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98dgh" podUID="e6e90e27-ad89-49df-904c-d28dd5ecaea4" Jul 21 00:16:53.291554 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:53.291522 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:16:53.291777 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:53.291651 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8sh6" podUID="982c215c-29b3-45ac-8525-03c61f27e931" Jul 21 00:16:53.292007 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:53.291990 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:16:53.292069 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:53.292060 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2psfz" podUID="23335158-224d-46e1-b617-10ffa4acb460" Jul 21 00:16:54.291673 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:54.291637 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:16:54.292069 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:54.291776 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98dgh" podUID="e6e90e27-ad89-49df-904c-d28dd5ecaea4" Jul 21 00:16:55.291874 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:55.291816 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:16:55.292294 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:55.291956 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8sh6" podUID="982c215c-29b3-45ac-8525-03c61f27e931" Jul 21 00:16:55.292331 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:55.292313 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:16:55.292433 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:55.292410 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2psfz" podUID="23335158-224d-46e1-b617-10ffa4acb460" Jul 21 00:16:55.868482 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:55.868432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs\") pod \"network-metrics-daemon-n8sh6\" (UID: \"982c215c-29b3-45ac-8525-03c61f27e931\") " pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:16:55.868700 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:55.868683 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:16:55.868774 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:55.868757 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs podName:982c215c-29b3-45ac-8525-03c61f27e931 nodeName:}" failed. No retries permitted until 2026-07-21 00:17:03.868735965 +0000 UTC m=+18.084116291 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs") pod "network-metrics-daemon-n8sh6" (UID: "982c215c-29b3-45ac-8525-03c61f27e931") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:16:55.970567 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:55.969738 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/23335158-224d-46e1-b617-10ffa4acb460-original-pull-secret\") pod \"global-pull-secret-syncer-2psfz\" (UID: \"23335158-224d-46e1-b617-10ffa4acb460\") " pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:16:55.970567 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:55.969832 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnbcl\" (UniqueName: \"kubernetes.io/projected/e6e90e27-ad89-49df-904c-d28dd5ecaea4-kube-api-access-tnbcl\") pod \"network-check-target-98dgh\" (UID: \"e6e90e27-ad89-49df-904c-d28dd5ecaea4\") " pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:16:55.970567 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:55.969993 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 00:16:55.970567 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:55.970013 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 00:16:55.970567 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:55.970026 2580 projected.go:196] Error preparing data for projected volume kube-api-access-tnbcl for pod openshift-network-diagnostics/network-check-target-98dgh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:16:55.970567 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:55.970085 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6e90e27-ad89-49df-904c-d28dd5ecaea4-kube-api-access-tnbcl podName:e6e90e27-ad89-49df-904c-d28dd5ecaea4 nodeName:}" failed. No retries permitted until 2026-07-21 00:17:03.970066725 +0000 UTC m=+18.185447046 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tnbcl" (UniqueName: "kubernetes.io/projected/e6e90e27-ad89-49df-904c-d28dd5ecaea4-kube-api-access-tnbcl") pod "network-check-target-98dgh" (UID: "e6e90e27-ad89-49df-904c-d28dd5ecaea4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:16:55.970567 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:55.970480 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 00:16:55.970567 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:55.970530 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23335158-224d-46e1-b617-10ffa4acb460-original-pull-secret podName:23335158-224d-46e1-b617-10ffa4acb460 nodeName:}" failed. No retries permitted until 2026-07-21 00:17:03.970516532 +0000 UTC m=+18.185896850 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/23335158-224d-46e1-b617-10ffa4acb460-original-pull-secret") pod "global-pull-secret-syncer-2psfz" (UID: "23335158-224d-46e1-b617-10ffa4acb460") : object "kube-system"/"original-pull-secret" not registered Jul 21 00:16:56.295181 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:56.294674 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:16:56.295181 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:56.294797 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98dgh" podUID="e6e90e27-ad89-49df-904c-d28dd5ecaea4" Jul 21 00:16:57.291450 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:57.291415 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:16:57.291643 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:57.291529 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8sh6" podUID="982c215c-29b3-45ac-8525-03c61f27e931" Jul 21 00:16:57.291942 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:57.291918 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:16:57.292048 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:57.292014 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2psfz" podUID="23335158-224d-46e1-b617-10ffa4acb460" Jul 21 00:16:58.291409 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:58.291370 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:16:58.291965 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:58.291501 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98dgh" podUID="e6e90e27-ad89-49df-904c-d28dd5ecaea4" Jul 21 00:16:59.291380 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:59.291343 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:16:59.291578 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:16:59.291343 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:16:59.291578 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:59.291457 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8sh6" podUID="982c215c-29b3-45ac-8525-03c61f27e931" Jul 21 00:16:59.291578 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:16:59.291539 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2psfz" podUID="23335158-224d-46e1-b617-10ffa4acb460" Jul 21 00:17:00.291709 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:00.291615 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:17:00.292104 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:00.291762 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98dgh" podUID="e6e90e27-ad89-49df-904c-d28dd5ecaea4" Jul 21 00:17:01.290958 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:01.290921 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:17:01.291149 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:01.291050 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8sh6" podUID="982c215c-29b3-45ac-8525-03c61f27e931" Jul 21 00:17:01.291149 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:01.291088 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:17:01.291254 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:01.291167 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2psfz" podUID="23335158-224d-46e1-b617-10ffa4acb460" Jul 21 00:17:02.291656 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:02.291609 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:17:02.292145 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:02.291732 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98dgh" podUID="e6e90e27-ad89-49df-904c-d28dd5ecaea4" Jul 21 00:17:03.291073 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:03.291032 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:17:03.291315 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:03.291043 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:17:03.291315 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:03.291161 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8sh6" podUID="982c215c-29b3-45ac-8525-03c61f27e931" Jul 21 00:17:03.291315 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:03.291206 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2psfz" podUID="23335158-224d-46e1-b617-10ffa4acb460" Jul 21 00:17:03.931062 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:03.931026 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs\") pod \"network-metrics-daemon-n8sh6\" (UID: \"982c215c-29b3-45ac-8525-03c61f27e931\") " pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:17:03.931532 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:03.931182 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:17:03.931532 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:03.931249 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs podName:982c215c-29b3-45ac-8525-03c61f27e931 nodeName:}" failed. No retries permitted until 2026-07-21 00:17:19.931226911 +0000 UTC m=+34.146607234 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs") pod "network-metrics-daemon-n8sh6" (UID: "982c215c-29b3-45ac-8525-03c61f27e931") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:17:04.032006 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:04.031968 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnbcl\" (UniqueName: \"kubernetes.io/projected/e6e90e27-ad89-49df-904c-d28dd5ecaea4-kube-api-access-tnbcl\") pod \"network-check-target-98dgh\" (UID: \"e6e90e27-ad89-49df-904c-d28dd5ecaea4\") " pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:17:04.032189 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:04.032027 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/23335158-224d-46e1-b617-10ffa4acb460-original-pull-secret\") pod \"global-pull-secret-syncer-2psfz\" (UID: \"23335158-224d-46e1-b617-10ffa4acb460\") " pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:17:04.032189 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:04.032154 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 00:17:04.032189 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:04.032159 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 00:17:04.032189 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:04.032187 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 00:17:04.032402 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:04.032198 2580 projected.go:196] Error preparing data for projected volume kube-api-access-tnbcl for pod openshift-network-diagnostics/network-check-target-98dgh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:17:04.032402 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:04.032227 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23335158-224d-46e1-b617-10ffa4acb460-original-pull-secret podName:23335158-224d-46e1-b617-10ffa4acb460 nodeName:}" failed. No retries permitted until 2026-07-21 00:17:20.032207456 +0000 UTC m=+34.247587777 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/23335158-224d-46e1-b617-10ffa4acb460-original-pull-secret") pod "global-pull-secret-syncer-2psfz" (UID: "23335158-224d-46e1-b617-10ffa4acb460") : object "kube-system"/"original-pull-secret" not registered Jul 21 00:17:04.032402 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:04.032248 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6e90e27-ad89-49df-904c-d28dd5ecaea4-kube-api-access-tnbcl podName:e6e90e27-ad89-49df-904c-d28dd5ecaea4 nodeName:}" failed. No retries permitted until 2026-07-21 00:17:20.0322344 +0000 UTC m=+34.247614705 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tnbcl" (UniqueName: "kubernetes.io/projected/e6e90e27-ad89-49df-904c-d28dd5ecaea4-kube-api-access-tnbcl") pod "network-check-target-98dgh" (UID: "e6e90e27-ad89-49df-904c-d28dd5ecaea4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:17:04.294008 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:04.293932 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:17:04.294156 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:04.294049 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98dgh" podUID="e6e90e27-ad89-49df-904c-d28dd5ecaea4" Jul 21 00:17:05.291804 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:05.291772 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:17:05.291804 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:05.291803 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:17:05.292314 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:05.291916 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8sh6" podUID="982c215c-29b3-45ac-8525-03c61f27e931" Jul 21 00:17:05.292314 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:05.291969 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2psfz" podUID="23335158-224d-46e1-b617-10ffa4acb460" Jul 21 00:17:06.291936 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:06.291899 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:17:06.292405 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:06.292013 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98dgh" podUID="e6e90e27-ad89-49df-904c-d28dd5ecaea4" Jul 21 00:17:07.291219 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:07.291173 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:17:07.291420 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:07.291179 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:17:07.291420 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:07.291300 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2psfz" podUID="23335158-224d-46e1-b617-10ffa4acb460" Jul 21 00:17:07.291420 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:07.291407 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8sh6" podUID="982c215c-29b3-45ac-8525-03c61f27e931" Jul 21 00:17:08.293945 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:08.293184 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:17:08.293945 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:08.293490 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98dgh" podUID="e6e90e27-ad89-49df-904c-d28dd5ecaea4" Jul 21 00:17:08.385076 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:08.385043 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fcl75" event={"ID":"6c602047-2c64-4c38-ba0d-8042a77f4e24","Type":"ContainerStarted","Data":"131bbf5be12378125067a105134a7030c167f9f8589c2d3ff629e6061b3d327e"} Jul 21 00:17:08.386694 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:08.386666 2580 generic.go:356] "Generic (PLEG): container finished" podID="5355db2a-b7df-48d3-9554-b5eaa4e937ab" containerID="4195b4f76e8488b89411f156d672a110c9747ad52b3785af03031cd2624088e3" exitCode=0 Jul 21 00:17:08.386833 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:08.386747 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k2vj4" event={"ID":"5355db2a-b7df-48d3-9554-b5eaa4e937ab","Type":"ContainerDied","Data":"4195b4f76e8488b89411f156d672a110c9747ad52b3785af03031cd2624088e3"} Jul 21 00:17:08.388735 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:08.388708 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p76tl" event={"ID":"2bd25715-f83d-42fa-953e-ba07d8b7da50","Type":"ContainerStarted","Data":"c76c25107a59d51be45081ce2af616ae031a111f1f8822c6c9e9fe7bfaf0f56e"} Jul 21 00:17:08.390535 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:08.390378 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jlx72" event={"ID":"da6931b3-d9b0-4c63-a0ba-7eca03b046db","Type":"ContainerStarted","Data":"a20ebc5f0be430cf18afdc01c5fb84fbf2c8e02808de1b5655f80bdf59f562b5"} Jul 21 00:17:08.392308 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:08.392279 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" event={"ID":"af62f4c5-4beb-44e6-903e-5c71ede9a3cd","Type":"ContainerStarted","Data":"3c7520ee9741688a1b54ede4dab914b56880927a429b025b4b62a733cbd820a3"} Jul 21 00:17:08.392426 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:08.392336 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" event={"ID":"af62f4c5-4beb-44e6-903e-5c71ede9a3cd","Type":"ContainerStarted","Data":"a9b3986eb8160e1aa9f68d7a380508eb2062a0999d836ad806064cd33e3421ce"} Jul 21 00:17:08.394322 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:08.394089 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-scmjq" event={"ID":"37628504-b773-4dad-a955-2a87d40ab71c","Type":"ContainerStarted","Data":"030c99537af938a264a917c8ba0f59f5684657e737565a162ff8bf258eeb8b3a"} Jul 21 00:17:08.395632 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:08.395562 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-szxbf" event={"ID":"583c4be0-d3e2-4a43-b91c-d6a8903569da","Type":"ContainerStarted","Data":"b3156f2e0fc4965ab39bb8162c1668957c329d0522477043b0ab9f2d81f1634c"} Jul 21 00:17:08.396997 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:08.396978 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" event={"ID":"48d1f8d1-615b-4b43-ab84-f2b22e43a382","Type":"ContainerStarted","Data":"e5c085870508a4bdad70be9913df912b4a253c9e6de23e4693aade4b7f2a9a50"} Jul 21 00:17:08.429225 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:08.429180 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-117.ec2.internal" podStartSLOduration=21.429149337 podStartE2EDuration="21.429149337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 00:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:16:51.365140914 +0000 UTC m=+5.580521241" watchObservedRunningTime="2026-07-21 00:17:08.429149337 +0000 UTC m=+22.644529655" Jul 21 00:17:08.454908 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:08.454864 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-scmjq" podStartSLOduration=3.504171677 podStartE2EDuration="22.454848648s" totalImagesPullingTime="18.950676971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 00:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:17:08.454653599 +0000 UTC m=+22.670033926" watchObservedRunningTime="2026-07-21 00:17:08.454848648 +0000 UTC m=+22.670228974" Jul 21 00:17:08.471678 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:08.471644 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jlx72" Jul 21 00:17:08.474340 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:08.474300 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fcl75" podStartSLOduration=3.547683203 podStartE2EDuration="22.47428558s" totalImagesPullingTime="18.926602377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 00:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:17:08.474203691 +0000 UTC m=+22.689584017" watchObservedRunningTime="2026-07-21 00:17:08.47428558 +0000 UTC m=+22.689666171" Jul 21 00:17:08.492626 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:08.492557 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-p76tl" podStartSLOduration=3.602103016 podStartE2EDuration="22.492544147s" totalImagesPullingTime="18.890441131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 00:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:17:08.491945911 +0000 UTC m=+22.707326237" watchObservedRunningTime="2026-07-21 00:17:08.492544147 +0000 UTC m=+22.707924473" Jul 21 00:17:08.530631 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:08.530567 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jlx72" podStartSLOduration=11.68976923 podStartE2EDuration="22.530553316s" totalImagesPullingTime="10.840784086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:17:08.511972407 +0000 UTC m=+22.727352734" watchObservedRunningTime="2026-07-21 00:17:08.530553316 +0000 UTC m=+22.745933642" Jul 21 00:17:08.530782 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:08.530708 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-szxbf" podStartSLOduration=3.64493178 podStartE2EDuration="22.530702885s" totalImagesPullingTime="18.885771105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 00:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:17:08.530289782 +0000 UTC m=+22.745670107" watchObservedRunningTime="2026-07-21 00:17:08.530702885 +0000 UTC m=+22.746083210" Jul 21 00:17:09.291250 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:09.291214 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:17:09.291422 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:09.291340 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8sh6" podUID="982c215c-29b3-45ac-8525-03c61f27e931" Jul 21 00:17:09.291422 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:09.291404 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:17:09.291553 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:09.291528 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2psfz" podUID="23335158-224d-46e1-b617-10ffa4acb460" Jul 21 00:17:09.404266 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:09.404185 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" event={"ID":"af62f4c5-4beb-44e6-903e-5c71ede9a3cd","Type":"ContainerStarted","Data":"fe313d6c5fa015d9fd21b3718f3b983e1860ee36722cce234fac0e87709da5dc"} Jul 21 00:17:09.404266 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:09.404274 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" event={"ID":"af62f4c5-4beb-44e6-903e-5c71ede9a3cd","Type":"ContainerStarted","Data":"7c73674e8edb451f4c06258addd391b5431231a558365c2f49e11a3edb6b095d"} Jul 21 00:17:09.405189 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:09.404327 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" event={"ID":"af62f4c5-4beb-44e6-903e-5c71ede9a3cd","Type":"ContainerStarted","Data":"b390a8924df59acbbe0c37a2fa9f73c0523aab62e9ac36e612bb1770ccd3efad"} Jul 21 00:17:09.405189 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:09.404439 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" event={"ID":"af62f4c5-4beb-44e6-903e-5c71ede9a3cd","Type":"ContainerStarted","Data":"b6634ff7d103b81af3ff8b4ecd6b3b30b93894dc91045ebacc6484356769de5b"} Jul 21 00:17:09.921569 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:09.921530 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jlx72" Jul 21 00:17:09.922423 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:09.922408 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jlx72" Jul 21 00:17:09.994537 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:09.994487 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 21 00:17:10.249857 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:10.249714 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T00:17:09.994512454Z","UUID":"2059595c-1183-4d02-9034-21c0be8b71e7","Handler":null,"Name":"","Endpoint":""} Jul 21 00:17:10.251288 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:10.251259 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 21 00:17:10.251288 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:10.251295 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 21 00:17:10.291030 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:10.291000 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:17:10.291196 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:10.291108 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98dgh" podUID="e6e90e27-ad89-49df-904c-d28dd5ecaea4" Jul 21 00:17:10.408354 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:10.408321 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" event={"ID":"48d1f8d1-615b-4b43-ab84-f2b22e43a382","Type":"ContainerStarted","Data":"e548a5f9d688733770f54f2bef0a50a508609329cbaa297c387a571c6c4e1b24"} Jul 21 00:17:10.409702 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:10.409678 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6pvbf" event={"ID":"00221678-4e87-4266-9c77-51182bb87c2b","Type":"ContainerStarted","Data":"096460ae7d32279a9a9606e09bf900c2a64630f269f13366598142ab80c244f1"} Jul 21 00:17:10.410282 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:10.410265 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jlx72" Jul 21 00:17:10.437478 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:10.437417 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6pvbf" podStartSLOduration=5.551313392 podStartE2EDuration="24.437398221s" totalImagesPullingTime="18.886084829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 00:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:17:10.423421521 +0000 UTC m=+24.638801847" watchObservedRunningTime="2026-07-21 00:17:10.437398221 +0000 UTC m=+24.652778540" Jul 21 00:17:11.291696 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:11.291663 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:17:11.291866 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:11.291676 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:17:11.291866 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:11.291775 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8sh6" podUID="982c215c-29b3-45ac-8525-03c61f27e931" Jul 21 00:17:11.291866 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:11.291854 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2psfz" podUID="23335158-224d-46e1-b617-10ffa4acb460" Jul 21 00:17:11.415405 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:11.415372 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" event={"ID":"af62f4c5-4beb-44e6-903e-5c71ede9a3cd","Type":"ContainerStarted","Data":"9c841f3106626f9eb3376172a0edd38155d7f328f3a0dbd8f73a3f838c0bdbc1"} Jul 21 00:17:11.417258 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:11.417237 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" event={"ID":"48d1f8d1-615b-4b43-ab84-f2b22e43a382","Type":"ContainerStarted","Data":"e337a6dca90a29a978fe066368a9a2c96214941aace1770376245c47568ca44d"} Jul 21 00:17:11.438436 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:11.438383 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rnjq8" podStartSLOduration=3.788363807 podStartE2EDuration="25.438368799s" totalImagesPullingTime="21.650004992s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 00:16:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:17:11.437803213 +0000 UTC m=+25.653183539" watchObservedRunningTime="2026-07-21 00:17:11.438368799 +0000 UTC m=+25.653749124" Jul 21 00:17:12.291834 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:12.291642 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:17:12.292037 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:12.291936 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98dgh" podUID="e6e90e27-ad89-49df-904c-d28dd5ecaea4" Jul 21 00:17:13.291858 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:13.291820 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:17:13.292369 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:13.291868 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:17:13.292369 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:13.291961 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8sh6" podUID="982c215c-29b3-45ac-8525-03c61f27e931" Jul 21 00:17:13.292369 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:13.292085 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2psfz" podUID="23335158-224d-46e1-b617-10ffa4acb460" Jul 21 00:17:14.291713 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:14.291680 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:17:14.291860 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:14.291812 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98dgh" podUID="e6e90e27-ad89-49df-904c-d28dd5ecaea4" Jul 21 00:17:14.426392 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:14.426343 2580 generic.go:356] "Generic (PLEG): container finished" podID="5355db2a-b7df-48d3-9554-b5eaa4e937ab" containerID="adaeb8832c7e43f23c2de1bbe6be5f16cc6283e9f18a6d6334494a4f0b0b6195" exitCode=0 Jul 21 00:17:14.427242 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:14.426390 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k2vj4" event={"ID":"5355db2a-b7df-48d3-9554-b5eaa4e937ab","Type":"ContainerDied","Data":"adaeb8832c7e43f23c2de1bbe6be5f16cc6283e9f18a6d6334494a4f0b0b6195"} Jul 21 00:17:14.430276 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:14.430247 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" event={"ID":"af62f4c5-4beb-44e6-903e-5c71ede9a3cd","Type":"ContainerStarted","Data":"0ce6db3d361263543f3453631299b456c2caa1da3416172eda9687d39bb8616b"} Jul 21 00:17:14.430513 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:14.430489 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:17:14.430639 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:14.430580 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:17:14.432096 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:14.432069 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:17:14.446836 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:14.446809 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:17:14.446995 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:14.446916 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:17:14.481483 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:14.481386 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" podStartSLOduration=9.510126955 podStartE2EDuration="28.481368147s" totalImagesPullingTime="18.971241192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 00:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:17:14.481212007 +0000 UTC m=+28.696592327" watchObservedRunningTime="2026-07-21 00:17:14.481368147 +0000 UTC m=+28.696748472" Jul 21 00:17:15.291701 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:15.291472 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:17:15.291956 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:15.291519 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:17:15.291956 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:15.291790 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2psfz" podUID="23335158-224d-46e1-b617-10ffa4acb460" Jul 21 00:17:15.291956 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:15.291886 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8sh6" podUID="982c215c-29b3-45ac-8525-03c61f27e931" Jul 21 00:17:16.045933 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:16.045895 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2psfz"] Jul 21 00:17:16.046406 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:16.045991 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:17:16.046406 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:16.046082 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2psfz" podUID="23335158-224d-46e1-b617-10ffa4acb460" Jul 21 00:17:16.048750 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:16.048720 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-98dgh"] Jul 21 00:17:16.048889 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:16.048836 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:17:16.048954 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:16.048934 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98dgh" podUID="e6e90e27-ad89-49df-904c-d28dd5ecaea4" Jul 21 00:17:16.049451 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:16.049424 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n8sh6"] Jul 21 00:17:16.049559 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:16.049510 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:17:16.049662 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:16.049590 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8sh6" podUID="982c215c-29b3-45ac-8525-03c61f27e931" Jul 21 00:17:16.436884 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:16.436851 2580 generic.go:356] "Generic (PLEG): container finished" podID="5355db2a-b7df-48d3-9554-b5eaa4e937ab" containerID="864b22e5e4120aaea30f35cc62fb5e5bc53ed6e5b6a894c34035a8259289ac28" exitCode=0 Jul 21 00:17:16.437064 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:16.436918 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k2vj4" event={"ID":"5355db2a-b7df-48d3-9554-b5eaa4e937ab","Type":"ContainerDied","Data":"864b22e5e4120aaea30f35cc62fb5e5bc53ed6e5b6a894c34035a8259289ac28"} Jul 21 00:17:17.291618 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:17.291569 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:17:17.291958 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:17.291700 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98dgh" podUID="e6e90e27-ad89-49df-904c-d28dd5ecaea4" Jul 21 00:17:17.411366 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:17.411334 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:17.504144 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:17.504066 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:18.291653 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:18.291625 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:17:18.292032 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:18.291734 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8sh6" podUID="982c215c-29b3-45ac-8525-03c61f27e931" Jul 21 00:17:18.292032 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:18.291787 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:17:18.292032 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:18.291878 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2psfz" podUID="23335158-224d-46e1-b617-10ffa4acb460" Jul 21 00:17:18.323774 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:18.323722 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:18.440910 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:18.440800 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:18.443777 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:18.443746 2580 generic.go:356] "Generic (PLEG): container finished" podID="5355db2a-b7df-48d3-9554-b5eaa4e937ab" containerID="c7f49898bbe59c0ae0106c955f1ef55ef87e627331ebd586f4c688c97f380bbb" exitCode=0 Jul 21 00:17:18.443911 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:18.443817 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k2vj4" event={"ID":"5355db2a-b7df-48d3-9554-b5eaa4e937ab","Type":"ContainerDied","Data":"c7f49898bbe59c0ae0106c955f1ef55ef87e627331ebd586f4c688c97f380bbb"} Jul 21 00:17:18.542944 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:18.542901 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:19.291852 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:19.291817 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:17:19.292406 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:19.291958 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98dgh" podUID="e6e90e27-ad89-49df-904c-d28dd5ecaea4" Jul 21 00:17:19.952483 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:19.952433 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs\") pod \"network-metrics-daemon-n8sh6\" (UID: \"982c215c-29b3-45ac-8525-03c61f27e931\") " pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:17:19.952667 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:19.952616 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:17:19.952727 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:19.952691 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs podName:982c215c-29b3-45ac-8525-03c61f27e931 nodeName:}" failed. No retries permitted until 2026-07-21 00:17:51.952670198 +0000 UTC m=+66.168050517 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs") pod "network-metrics-daemon-n8sh6" (UID: "982c215c-29b3-45ac-8525-03c61f27e931") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 00:17:20.053381 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:20.053296 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnbcl\" (UniqueName: \"kubernetes.io/projected/e6e90e27-ad89-49df-904c-d28dd5ecaea4-kube-api-access-tnbcl\") pod \"network-check-target-98dgh\" (UID: \"e6e90e27-ad89-49df-904c-d28dd5ecaea4\") " pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:17:20.053381 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:20.053359 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/23335158-224d-46e1-b617-10ffa4acb460-original-pull-secret\") pod \"global-pull-secret-syncer-2psfz\" (UID: \"23335158-224d-46e1-b617-10ffa4acb460\") " pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:17:20.053584 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:20.053471 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 00:17:20.053584 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:20.053476 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 00:17:20.053584 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:20.053500 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 00:17:20.053584 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:20.053514 2580 projected.go:196] Error preparing data for projected volume kube-api-access-tnbcl for pod openshift-network-diagnostics/network-check-target-98dgh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:17:20.053584 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:20.053532 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23335158-224d-46e1-b617-10ffa4acb460-original-pull-secret podName:23335158-224d-46e1-b617-10ffa4acb460 nodeName:}" failed. No retries permitted until 2026-07-21 00:17:52.053513317 +0000 UTC m=+66.268893641 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/23335158-224d-46e1-b617-10ffa4acb460-original-pull-secret") pod "global-pull-secret-syncer-2psfz" (UID: "23335158-224d-46e1-b617-10ffa4acb460") : object "kube-system"/"original-pull-secret" not registered Jul 21 00:17:20.053584 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:20.053564 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6e90e27-ad89-49df-904c-d28dd5ecaea4-kube-api-access-tnbcl podName:e6e90e27-ad89-49df-904c-d28dd5ecaea4 nodeName:}" failed. No retries permitted until 2026-07-21 00:17:52.053551724 +0000 UTC m=+66.268932044 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tnbcl" (UniqueName: "kubernetes.io/projected/e6e90e27-ad89-49df-904c-d28dd5ecaea4-kube-api-access-tnbcl") pod "network-check-target-98dgh" (UID: "e6e90e27-ad89-49df-904c-d28dd5ecaea4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 00:17:20.291621 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:20.291570 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:17:20.291817 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:20.291740 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2psfz" podUID="23335158-224d-46e1-b617-10ffa4acb460" Jul 21 00:17:20.291817 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:20.291798 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:17:20.292236 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:20.291930 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n8sh6" podUID="982c215c-29b3-45ac-8525-03c61f27e931" Jul 21 00:17:21.291525 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:21.291487 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:17:21.291734 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:21.291622 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98dgh" podUID="e6e90e27-ad89-49df-904c-d28dd5ecaea4" Jul 21 00:17:22.039683 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.039332 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-117.ec2.internal" event="NodeReady" Jul 21 00:17:22.039683 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.039488 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 21 00:17:22.086259 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.086227 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pz4d4"] Jul 21 00:17:22.089133 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.089112 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-br7vh"] Jul 21 00:17:22.089274 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.089245 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pz4d4" Jul 21 00:17:22.091071 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.091054 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-br7vh" Jul 21 00:17:22.092153 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.092126 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 21 00:17:22.092264 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.092224 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 21 00:17:22.092856 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.092828 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-26lqt\"" Jul 21 00:17:22.093761 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.093546 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 21 00:17:22.093761 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.093681 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 21 00:17:22.095750 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.095727 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 21 00:17:22.096126 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.095937 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-znvqg\"" Jul 21 00:17:22.097787 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.097698 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-br7vh"] Jul 21 00:17:22.101540 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.099445 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pz4d4"] Jul 21 00:17:22.214718 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.214683 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:22.268966 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.268932 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5046e73d-17d6-4282-9387-c665c60f7ca9-cert\") pod \"ingress-canary-br7vh\" (UID: \"5046e73d-17d6-4282-9387-c665c60f7ca9\") " pod="openshift-ingress-canary/ingress-canary-br7vh" Jul 21 00:17:22.269149 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.269007 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5lt4\" (UniqueName: \"kubernetes.io/projected/59620b96-9827-4757-841e-be5c630ef424-kube-api-access-f5lt4\") pod \"dns-default-pz4d4\" (UID: \"59620b96-9827-4757-841e-be5c630ef424\") " pod="openshift-dns/dns-default-pz4d4" Jul 21 00:17:22.269149 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.269114 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/59620b96-9827-4757-841e-be5c630ef424-config-volume\") pod \"dns-default-pz4d4\" (UID: \"59620b96-9827-4757-841e-be5c630ef424\") " pod="openshift-dns/dns-default-pz4d4" Jul 21 00:17:22.269242 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.269149 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls\") pod \"dns-default-pz4d4\" (UID: \"59620b96-9827-4757-841e-be5c630ef424\") " pod="openshift-dns/dns-default-pz4d4" Jul 21 00:17:22.269242 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.269195 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59620b96-9827-4757-841e-be5c630ef424-tmp-dir\") pod \"dns-default-pz4d4\" (UID: \"59620b96-9827-4757-841e-be5c630ef424\") " pod="openshift-dns/dns-default-pz4d4" Jul 21 00:17:22.269242 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.269220 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh97s\" (UniqueName: \"kubernetes.io/projected/5046e73d-17d6-4282-9387-c665c60f7ca9-kube-api-access-vh97s\") pod \"ingress-canary-br7vh\" (UID: \"5046e73d-17d6-4282-9387-c665c60f7ca9\") " pod="openshift-ingress-canary/ingress-canary-br7vh" Jul 21 00:17:22.291512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.291434 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:17:22.291699 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.291663 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:17:22.294506 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.294481 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 00:17:22.294651 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.294532 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 00:17:22.294651 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.294574 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ztz7f\"" Jul 21 00:17:22.298634 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.298577 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:22.370019 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.369979 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5lt4\" (UniqueName: \"kubernetes.io/projected/59620b96-9827-4757-841e-be5c630ef424-kube-api-access-f5lt4\") pod \"dns-default-pz4d4\" (UID: \"59620b96-9827-4757-841e-be5c630ef424\") " pod="openshift-dns/dns-default-pz4d4" Jul 21 00:17:22.370176 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.370028 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/59620b96-9827-4757-841e-be5c630ef424-config-volume\") pod \"dns-default-pz4d4\" (UID: \"59620b96-9827-4757-841e-be5c630ef424\") " pod="openshift-dns/dns-default-pz4d4" Jul 21 00:17:22.370176 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.370047 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls\") pod \"dns-default-pz4d4\" (UID: \"59620b96-9827-4757-841e-be5c630ef424\") " pod="openshift-dns/dns-default-pz4d4" Jul 21 00:17:22.370176 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.370067 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59620b96-9827-4757-841e-be5c630ef424-tmp-dir\") pod \"dns-default-pz4d4\" (UID: \"59620b96-9827-4757-841e-be5c630ef424\") " pod="openshift-dns/dns-default-pz4d4" Jul 21 00:17:22.370176 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.370082 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh97s\" (UniqueName: \"kubernetes.io/projected/5046e73d-17d6-4282-9387-c665c60f7ca9-kube-api-access-vh97s\") pod \"ingress-canary-br7vh\" (UID: \"5046e73d-17d6-4282-9387-c665c60f7ca9\") " pod="openshift-ingress-canary/ingress-canary-br7vh" Jul 21 00:17:22.370176 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.370104 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5046e73d-17d6-4282-9387-c665c60f7ca9-cert\") pod \"ingress-canary-br7vh\" (UID: \"5046e73d-17d6-4282-9387-c665c60f7ca9\") " pod="openshift-ingress-canary/ingress-canary-br7vh" Jul 21 00:17:22.370176 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:22.370154 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 00:17:22.370471 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:22.370230 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls podName:59620b96-9827-4757-841e-be5c630ef424 nodeName:}" failed. No retries permitted until 2026-07-21 00:17:22.870209612 +0000 UTC m=+37.085589917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls") pod "dns-default-pz4d4" (UID: "59620b96-9827-4757-841e-be5c630ef424") : secret "dns-default-metrics-tls" not found Jul 21 00:17:22.370471 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.370411 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59620b96-9827-4757-841e-be5c630ef424-tmp-dir\") pod \"dns-default-pz4d4\" (UID: \"59620b96-9827-4757-841e-be5c630ef424\") " pod="openshift-dns/dns-default-pz4d4" Jul 21 00:17:22.370580 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.370564 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/59620b96-9827-4757-841e-be5c630ef424-config-volume\") pod \"dns-default-pz4d4\" (UID: \"59620b96-9827-4757-841e-be5c630ef424\") " pod="openshift-dns/dns-default-pz4d4" Jul 21 00:17:22.374745 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.374721 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5046e73d-17d6-4282-9387-c665c60f7ca9-cert\") pod \"ingress-canary-br7vh\" (UID: \"5046e73d-17d6-4282-9387-c665c60f7ca9\") " pod="openshift-ingress-canary/ingress-canary-br7vh" Jul 21 00:17:22.395258 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.395222 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5lt4\" (UniqueName: \"kubernetes.io/projected/59620b96-9827-4757-841e-be5c630ef424-kube-api-access-f5lt4\") pod \"dns-default-pz4d4\" (UID: \"59620b96-9827-4757-841e-be5c630ef424\") " pod="openshift-dns/dns-default-pz4d4" Jul 21 00:17:22.395425 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.395405 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh97s\" (UniqueName: \"kubernetes.io/projected/5046e73d-17d6-4282-9387-c665c60f7ca9-kube-api-access-vh97s\") pod \"ingress-canary-br7vh\" (UID: \"5046e73d-17d6-4282-9387-c665c60f7ca9\") " pod="openshift-ingress-canary/ingress-canary-br7vh" Jul 21 00:17:22.410402 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.410357 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-br7vh" Jul 21 00:17:22.433662 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.433630 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:22.606875 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.606681 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-br7vh"] Jul 21 00:17:22.610089 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:17:22.610057 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5046e73d_17d6_4282_9387_c665c60f7ca9.slice/crio-a556b01ff3191c67a5e0352c3b45325c0bef9fa20a51e639ccc5249b8cff1c21 WatchSource:0}: Error finding container a556b01ff3191c67a5e0352c3b45325c0bef9fa20a51e639ccc5249b8cff1c21: Status 404 returned error can't find the container with id a556b01ff3191c67a5e0352c3b45325c0bef9fa20a51e639ccc5249b8cff1c21 Jul 21 00:17:22.872837 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:22.872801 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls\") pod \"dns-default-pz4d4\" (UID: \"59620b96-9827-4757-841e-be5c630ef424\") " pod="openshift-dns/dns-default-pz4d4" Jul 21 00:17:22.873001 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:22.872963 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 00:17:22.873044 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:22.873034 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls podName:59620b96-9827-4757-841e-be5c630ef424 nodeName:}" failed. No retries permitted until 2026-07-21 00:17:23.873013452 +0000 UTC m=+38.088393781 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls") pod "dns-default-pz4d4" (UID: "59620b96-9827-4757-841e-be5c630ef424") : secret "dns-default-metrics-tls" not found Jul 21 00:17:23.291374 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:23.291296 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:17:23.298633 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:23.295084 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 00:17:23.298633 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:23.295194 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-m4mrq\"" Jul 21 00:17:23.298633 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:23.296153 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 00:17:23.458578 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:23.458537 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-br7vh" event={"ID":"5046e73d-17d6-4282-9387-c665c60f7ca9","Type":"ContainerStarted","Data":"a556b01ff3191c67a5e0352c3b45325c0bef9fa20a51e639ccc5249b8cff1c21"} Jul 21 00:17:23.461883 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:23.461855 2580 generic.go:356] "Generic (PLEG): container finished" podID="5355db2a-b7df-48d3-9554-b5eaa4e937ab" containerID="4101ec71bc5c5f83f425e00fdcc6af584b73fd5108ea1ce73624d819777883a0" exitCode=0 Jul 21 00:17:23.462025 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:23.461907 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k2vj4" event={"ID":"5355db2a-b7df-48d3-9554-b5eaa4e937ab","Type":"ContainerDied","Data":"4101ec71bc5c5f83f425e00fdcc6af584b73fd5108ea1ce73624d819777883a0"} Jul 21 00:17:23.880492 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:23.880392 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls\") pod \"dns-default-pz4d4\" (UID: \"59620b96-9827-4757-841e-be5c630ef424\") " pod="openshift-dns/dns-default-pz4d4" Jul 21 00:17:23.880684 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:23.880494 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 00:17:23.880684 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:23.880570 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls podName:59620b96-9827-4757-841e-be5c630ef424 nodeName:}" failed. No retries permitted until 2026-07-21 00:17:25.880550023 +0000 UTC m=+40.095930331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls") pod "dns-default-pz4d4" (UID: "59620b96-9827-4757-841e-be5c630ef424") : secret "dns-default-metrics-tls" not found Jul 21 00:17:24.468406 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:24.468367 2580 generic.go:356] "Generic (PLEG): container finished" podID="5355db2a-b7df-48d3-9554-b5eaa4e937ab" containerID="68f45e9aff9a5bf2c1827e1b4672f7372596da9593601b41c25fede5bfd2e226" exitCode=0 Jul 21 00:17:24.468887 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:24.468420 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k2vj4" event={"ID":"5355db2a-b7df-48d3-9554-b5eaa4e937ab","Type":"ContainerDied","Data":"68f45e9aff9a5bf2c1827e1b4672f7372596da9593601b41c25fede5bfd2e226"} Jul 21 00:17:25.264619 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:25.264335 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:25.318427 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:25.318388 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:25.472880 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:25.472836 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-br7vh" event={"ID":"5046e73d-17d6-4282-9387-c665c60f7ca9","Type":"ContainerStarted","Data":"2bc3bd3573ed05a7c529408d66e470bcf95b94a635a40d0c424e1d7c11c34d93"} Jul 21 00:17:25.476031 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:25.476002 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k2vj4" event={"ID":"5355db2a-b7df-48d3-9554-b5eaa4e937ab","Type":"ContainerStarted","Data":"f5a9cae5d007656df8d337d0998e47a0801178011284f1fb3a1889914f190501"} Jul 21 00:17:25.491222 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:25.491168 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-br7vh" podStartSLOduration=1.19053321 podStartE2EDuration="3.491150419s" totalImagesPullingTime="2.300617209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:17:25.490129376 +0000 UTC m=+39.705509747" watchObservedRunningTime="2026-07-21 00:17:25.491150419 +0000 UTC m=+39.706530744" Jul 21 00:17:25.516550 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:25.516445 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k2vj4" podStartSLOduration=8.66658234 podStartE2EDuration="39.516426108s" totalImagesPullingTime="30.849843768s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 00:16:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:17:25.514525336 +0000 UTC m=+39.729905661" watchObservedRunningTime="2026-07-21 00:17:25.516426108 +0000 UTC m=+39.731806433" Jul 21 00:17:25.640387 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:25.640341 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:25.707966 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:25.707929 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:25.898102 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:25.898052 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls\") pod \"dns-default-pz4d4\" (UID: \"59620b96-9827-4757-841e-be5c630ef424\") " pod="openshift-dns/dns-default-pz4d4" Jul 21 00:17:25.898272 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:25.898207 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 00:17:25.898312 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:25.898273 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls podName:59620b96-9827-4757-841e-be5c630ef424 nodeName:}" failed. No retries permitted until 2026-07-21 00:17:29.898257067 +0000 UTC m=+44.113637372 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls") pod "dns-default-pz4d4" (UID: "59620b96-9827-4757-841e-be5c630ef424") : secret "dns-default-metrics-tls" not found Jul 21 00:17:26.425655 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:26.425621 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:26.487937 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:26.487893 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:28.892978 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:28.892933 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:28.964897 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:28.964867 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:29.652510 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:29.652477 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:29.705789 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:29.705750 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:29.928186 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:29.928103 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls\") pod \"dns-default-pz4d4\" (UID: \"59620b96-9827-4757-841e-be5c630ef424\") " pod="openshift-dns/dns-default-pz4d4" Jul 21 00:17:29.928545 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:29.928265 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 00:17:29.928545 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:29.928330 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls podName:59620b96-9827-4757-841e-be5c630ef424 nodeName:}" failed. No retries permitted until 2026-07-21 00:17:37.92831384 +0000 UTC m=+52.143694144 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls") pod "dns-default-pz4d4" (UID: "59620b96-9827-4757-841e-be5c630ef424") : secret "dns-default-metrics-tls" not found Jul 21 00:17:35.344353 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:35.344307 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:35.440115 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:35.440074 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:35.544311 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:35.544283 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:35.646543 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:35.646458 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:37.983688 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:37.983637 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls\") pod \"dns-default-pz4d4\" (UID: \"59620b96-9827-4757-841e-be5c630ef424\") " pod="openshift-dns/dns-default-pz4d4" Jul 21 00:17:37.984061 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:37.983801 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 00:17:37.984061 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:37.983880 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls podName:59620b96-9827-4757-841e-be5c630ef424 nodeName:}" failed. No retries permitted until 2026-07-21 00:17:53.983861457 +0000 UTC m=+68.199241782 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls") pod "dns-default-pz4d4" (UID: "59620b96-9827-4757-841e-be5c630ef424") : secret "dns-default-metrics-tls" not found Jul 21 00:17:44.274581 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:44.274545 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:44.337641 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:44.337592 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:46.448203 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:46.448175 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mzgzs" Jul 21 00:17:46.606684 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:46.606645 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:46.681515 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:46.681481 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:49.308209 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:49.308177 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:51.971648 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:51.971593 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs\") pod \"network-metrics-daemon-n8sh6\" (UID: \"982c215c-29b3-45ac-8525-03c61f27e931\") " pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:17:51.974742 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:51.974723 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 00:17:51.982142 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:51.982124 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 21 00:17:51.982220 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:51.982189 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs podName:982c215c-29b3-45ac-8525-03c61f27e931 nodeName:}" failed. No retries permitted until 2026-07-21 00:18:55.982173004 +0000 UTC m=+130.197553308 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs") pod "network-metrics-daemon-n8sh6" (UID: "982c215c-29b3-45ac-8525-03c61f27e931") : secret "metrics-daemon-secret" not found Jul 21 00:17:52.072035 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:52.072000 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnbcl\" (UniqueName: \"kubernetes.io/projected/e6e90e27-ad89-49df-904c-d28dd5ecaea4-kube-api-access-tnbcl\") pod \"network-check-target-98dgh\" (UID: \"e6e90e27-ad89-49df-904c-d28dd5ecaea4\") " pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:17:52.072174 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:52.072064 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/23335158-224d-46e1-b617-10ffa4acb460-original-pull-secret\") pod \"global-pull-secret-syncer-2psfz\" (UID: \"23335158-224d-46e1-b617-10ffa4acb460\") " pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:17:52.074807 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:52.074788 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 00:17:52.074910 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:52.074809 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 00:17:52.085342 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:52.085320 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 00:17:52.086927 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:52.086905 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/23335158-224d-46e1-b617-10ffa4acb460-original-pull-secret\") pod \"global-pull-secret-syncer-2psfz\" (UID: \"23335158-224d-46e1-b617-10ffa4acb460\") " pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:17:52.095889 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:52.095859 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnbcl\" (UniqueName: \"kubernetes.io/projected/e6e90e27-ad89-49df-904c-d28dd5ecaea4-kube-api-access-tnbcl\") pod \"network-check-target-98dgh\" (UID: \"e6e90e27-ad89-49df-904c-d28dd5ecaea4\") " pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:17:52.107181 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:52.107152 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-m4mrq\"" Jul 21 00:17:52.115376 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:52.115352 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:17:52.249798 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:52.249720 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-98dgh"] Jul 21 00:17:52.253254 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:17:52.253221 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e90e27_ad89_49df_904c_d28dd5ecaea4.slice/crio-67f3e1e179d69500dfaebfbe6dcaa5735fd7bb8e0fdfe009f2ea323f84a15930 WatchSource:0}: Error finding container 67f3e1e179d69500dfaebfbe6dcaa5735fd7bb8e0fdfe009f2ea323f84a15930: Status 404 returned error can't find the container with id 67f3e1e179d69500dfaebfbe6dcaa5735fd7bb8e0fdfe009f2ea323f84a15930 Jul 21 00:17:52.304357 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:52.304328 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2psfz" Jul 21 00:17:52.429182 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:52.429147 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2psfz"] Jul 21 00:17:52.434209 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:17:52.434178 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23335158_224d_46e1_b617_10ffa4acb460.slice/crio-3bc9cd7e5a072a93395075f9ecc146228d19664be9abb96b51566072ac1fffff WatchSource:0}: Error finding container 3bc9cd7e5a072a93395075f9ecc146228d19664be9abb96b51566072ac1fffff: Status 404 returned error can't find the container with id 3bc9cd7e5a072a93395075f9ecc146228d19664be9abb96b51566072ac1fffff Jul 21 00:17:52.540384 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:52.540278 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-98dgh" event={"ID":"e6e90e27-ad89-49df-904c-d28dd5ecaea4","Type":"ContainerStarted","Data":"67f3e1e179d69500dfaebfbe6dcaa5735fd7bb8e0fdfe009f2ea323f84a15930"} Jul 21 00:17:52.541333 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:52.541313 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2psfz" event={"ID":"23335158-224d-46e1-b617-10ffa4acb460","Type":"ContainerStarted","Data":"3bc9cd7e5a072a93395075f9ecc146228d19664be9abb96b51566072ac1fffff"} Jul 21 00:17:53.985283 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:53.985243 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls\") pod \"dns-default-pz4d4\" (UID: \"59620b96-9827-4757-841e-be5c630ef424\") " pod="openshift-dns/dns-default-pz4d4" Jul 21 00:17:53.985884 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:53.985365 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 00:17:53.985884 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:17:53.985431 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls podName:59620b96-9827-4757-841e-be5c630ef424 nodeName:}" failed. No retries permitted until 2026-07-21 00:18:25.985411066 +0000 UTC m=+100.200791372 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls") pod "dns-default-pz4d4" (UID: "59620b96-9827-4757-841e-be5c630ef424") : secret "dns-default-metrics-tls" not found Jul 21 00:17:55.425970 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:55.425934 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:55.524170 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:55.524090 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:55.658590 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:55.658474 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:17:57.556536 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:57.556492 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2psfz" event={"ID":"23335158-224d-46e1-b617-10ffa4acb460","Type":"ContainerStarted","Data":"236626dd44a6e939bdf4f0512660829af3d4a59100d9806b66684589e46ec401"} Jul 21 00:17:57.557913 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:57.557890 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-98dgh" event={"ID":"e6e90e27-ad89-49df-904c-d28dd5ecaea4","Type":"ContainerStarted","Data":"70643fecfa3c0d7285563d43cac17bd7e308258412b4bf087baf57f768196ea7"} Jul 21 00:17:57.558029 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:57.558018 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:17:57.591835 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:57.591782 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2psfz" podStartSLOduration=66.989586732 podStartE2EDuration="1m11.591763133s" totalImagesPullingTime="4.602176401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 00:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:17:57.576243993 +0000 UTC m=+71.791624323" watchObservedRunningTime="2026-07-21 00:17:57.591763133 +0000 UTC m=+71.807143455" Jul 21 00:17:57.592029 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:17:57.592010 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-98dgh" podStartSLOduration=66.81183707 podStartE2EDuration="1m11.592003602s" totalImagesPullingTime="4.780166532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:17:57.591501606 +0000 UTC m=+71.806881931" watchObservedRunningTime="2026-07-21 00:17:57.592003602 +0000 UTC m=+71.807383927" Jul 21 00:18:19.530284 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:19.530143 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:18:22.564650 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.564616 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n"] Jul 21 00:18:22.567455 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.567437 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" Jul 21 00:18:22.570457 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.570432 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 21 00:18:22.571698 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.571679 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 21 00:18:22.571957 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.571940 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 21 00:18:22.572022 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.571961 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 21 00:18:22.572022 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.571995 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 21 00:18:22.572340 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.572318 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 21 00:18:22.572441 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.572318 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 21 00:18:22.585149 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.585121 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n"] Jul 21 00:18:22.673006 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.672967 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62cdn\" (UniqueName: \"kubernetes.io/projected/2a83348f-884c-40d7-b24a-77deeadc39ff-kube-api-access-62cdn\") pod \"cluster-proxy-proxy-agent-546885c98-xkf2n\" (UID: \"2a83348f-884c-40d7-b24a-77deeadc39ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" Jul 21 00:18:22.673006 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.673011 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a83348f-884c-40d7-b24a-77deeadc39ff-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-546885c98-xkf2n\" (UID: \"2a83348f-884c-40d7-b24a-77deeadc39ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" Jul 21 00:18:22.673232 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.673032 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2a83348f-884c-40d7-b24a-77deeadc39ff-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-546885c98-xkf2n\" (UID: \"2a83348f-884c-40d7-b24a-77deeadc39ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" Jul 21 00:18:22.673232 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.673087 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2a83348f-884c-40d7-b24a-77deeadc39ff-ca\") pod \"cluster-proxy-proxy-agent-546885c98-xkf2n\" (UID: \"2a83348f-884c-40d7-b24a-77deeadc39ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" Jul 21 00:18:22.673232 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.673112 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a83348f-884c-40d7-b24a-77deeadc39ff-additional-service-ca\") pod \"cluster-proxy-proxy-agent-546885c98-xkf2n\" (UID: \"2a83348f-884c-40d7-b24a-77deeadc39ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" Jul 21 00:18:22.673232 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.673153 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2a83348f-884c-40d7-b24a-77deeadc39ff-hub\") pod \"cluster-proxy-proxy-agent-546885c98-xkf2n\" (UID: \"2a83348f-884c-40d7-b24a-77deeadc39ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" Jul 21 00:18:22.774393 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.774363 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62cdn\" (UniqueName: \"kubernetes.io/projected/2a83348f-884c-40d7-b24a-77deeadc39ff-kube-api-access-62cdn\") pod \"cluster-proxy-proxy-agent-546885c98-xkf2n\" (UID: \"2a83348f-884c-40d7-b24a-77deeadc39ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" Jul 21 00:18:22.774516 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.774400 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a83348f-884c-40d7-b24a-77deeadc39ff-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-546885c98-xkf2n\" (UID: \"2a83348f-884c-40d7-b24a-77deeadc39ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" Jul 21 00:18:22.774516 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.774418 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2a83348f-884c-40d7-b24a-77deeadc39ff-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-546885c98-xkf2n\" (UID: \"2a83348f-884c-40d7-b24a-77deeadc39ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" Jul 21 00:18:22.774516 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.774453 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2a83348f-884c-40d7-b24a-77deeadc39ff-ca\") pod \"cluster-proxy-proxy-agent-546885c98-xkf2n\" (UID: \"2a83348f-884c-40d7-b24a-77deeadc39ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" Jul 21 00:18:22.774516 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.774480 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a83348f-884c-40d7-b24a-77deeadc39ff-additional-service-ca\") pod \"cluster-proxy-proxy-agent-546885c98-xkf2n\" (UID: \"2a83348f-884c-40d7-b24a-77deeadc39ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" Jul 21 00:18:22.774516 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.774506 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2a83348f-884c-40d7-b24a-77deeadc39ff-hub\") pod \"cluster-proxy-proxy-agent-546885c98-xkf2n\" (UID: \"2a83348f-884c-40d7-b24a-77deeadc39ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" Jul 21 00:18:22.775539 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.775508 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a83348f-884c-40d7-b24a-77deeadc39ff-additional-service-ca\") pod \"cluster-proxy-proxy-agent-546885c98-xkf2n\" (UID: \"2a83348f-884c-40d7-b24a-77deeadc39ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" Jul 21 00:18:22.777154 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.777116 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2a83348f-884c-40d7-b24a-77deeadc39ff-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-546885c98-xkf2n\" (UID: \"2a83348f-884c-40d7-b24a-77deeadc39ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" Jul 21 00:18:22.777249 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.777202 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2a83348f-884c-40d7-b24a-77deeadc39ff-ca\") pod \"cluster-proxy-proxy-agent-546885c98-xkf2n\" (UID: \"2a83348f-884c-40d7-b24a-77deeadc39ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" Jul 21 00:18:22.777351 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.777331 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2a83348f-884c-40d7-b24a-77deeadc39ff-hub\") pod \"cluster-proxy-proxy-agent-546885c98-xkf2n\" (UID: \"2a83348f-884c-40d7-b24a-77deeadc39ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" Jul 21 00:18:22.777415 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.777399 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2a83348f-884c-40d7-b24a-77deeadc39ff-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-546885c98-xkf2n\" (UID: \"2a83348f-884c-40d7-b24a-77deeadc39ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" Jul 21 00:18:22.783340 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.783315 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62cdn\" (UniqueName: \"kubernetes.io/projected/2a83348f-884c-40d7-b24a-77deeadc39ff-kube-api-access-62cdn\") pod \"cluster-proxy-proxy-agent-546885c98-xkf2n\" (UID: \"2a83348f-884c-40d7-b24a-77deeadc39ff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" Jul 21 00:18:22.889102 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:22.889004 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" Jul 21 00:18:23.017718 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:23.017694 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n"] Jul 21 00:18:23.020099 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:18:23.020072 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a83348f_884c_40d7_b24a_77deeadc39ff.slice/crio-2066134bc86a46a5aff08e690d82b642fa817cfa291c1f2dc3defc30f131576d WatchSource:0}: Error finding container 2066134bc86a46a5aff08e690d82b642fa817cfa291c1f2dc3defc30f131576d: Status 404 returned error can't find the container with id 2066134bc86a46a5aff08e690d82b642fa817cfa291c1f2dc3defc30f131576d Jul 21 00:18:23.620916 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:23.620873 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" event={"ID":"2a83348f-884c-40d7-b24a-77deeadc39ff","Type":"ContainerStarted","Data":"2066134bc86a46a5aff08e690d82b642fa817cfa291c1f2dc3defc30f131576d"} Jul 21 00:18:26.000903 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:26.000810 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls\") pod \"dns-default-pz4d4\" (UID: \"59620b96-9827-4757-841e-be5c630ef424\") " pod="openshift-dns/dns-default-pz4d4" Jul 21 00:18:26.001352 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:18:26.000968 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 00:18:26.001352 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:18:26.001070 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls podName:59620b96-9827-4757-841e-be5c630ef424 nodeName:}" failed. No retries permitted until 2026-07-21 00:19:30.001044733 +0000 UTC m=+164.216425076 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls") pod "dns-default-pz4d4" (UID: "59620b96-9827-4757-841e-be5c630ef424") : secret "dns-default-metrics-tls" not found Jul 21 00:18:26.631003 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:26.630912 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" event={"ID":"2a83348f-884c-40d7-b24a-77deeadc39ff","Type":"ContainerStarted","Data":"fc10e821b61177ba5daeca5d96203a023813164846afa6487b42b1b9ee573ad2"} Jul 21 00:18:26.631003 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:26.630963 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" event={"ID":"2a83348f-884c-40d7-b24a-77deeadc39ff","Type":"ContainerStarted","Data":"4a8b53d55656e74bf4944865a0b0427473e1ee867000cb66e5d71bb508c42e49"} Jul 21 00:18:26.631003 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:26.630981 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" event={"ID":"2a83348f-884c-40d7-b24a-77deeadc39ff","Type":"ContainerStarted","Data":"6513c8caa2dfba2b4758ca9c192531cdfadc7ebd47c0c86bf076c2f048f2be47"} Jul 21 00:18:26.653584 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:26.653534 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" podStartSLOduration=1.357296917 podStartE2EDuration="4.653518229s" totalImagesPullingTime="3.296221312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:18:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:18:26.652329945 +0000 UTC m=+100.867710272" watchObservedRunningTime="2026-07-21 00:18:26.653518229 +0000 UTC m=+100.868898533" Jul 21 00:18:28.562199 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:28.562168 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-98dgh" Jul 21 00:18:56.003738 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:18:56.003699 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs\") pod \"network-metrics-daemon-n8sh6\" (UID: \"982c215c-29b3-45ac-8525-03c61f27e931\") " pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:18:56.004149 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:18:56.003853 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 21 00:18:56.004149 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:18:56.003942 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs podName:982c215c-29b3-45ac-8525-03c61f27e931 nodeName:}" failed. No retries permitted until 2026-07-21 00:20:58.003925968 +0000 UTC m=+252.219306272 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs") pod "network-metrics-daemon-n8sh6" (UID: "982c215c-29b3-45ac-8525-03c61f27e931") : secret "metrics-daemon-secret" not found Jul 21 00:19:25.104411 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:19:25.104357 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pz4d4" podUID="59620b96-9827-4757-841e-be5c630ef424" Jul 21 00:19:25.310726 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:19:25.310686 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-n8sh6" podUID="982c215c-29b3-45ac-8525-03c61f27e931" Jul 21 00:19:25.790985 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:19:25.790952 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pz4d4" Jul 21 00:19:30.045620 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:19:30.045555 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls\") pod \"dns-default-pz4d4\" (UID: \"59620b96-9827-4757-841e-be5c630ef424\") " pod="openshift-dns/dns-default-pz4d4" Jul 21 00:19:30.046021 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:19:30.045714 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 00:19:30.046021 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:19:30.045788 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls podName:59620b96-9827-4757-841e-be5c630ef424 nodeName:}" failed. No retries permitted until 2026-07-21 00:21:32.045772984 +0000 UTC m=+286.261153288 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls") pod "dns-default-pz4d4" (UID: "59620b96-9827-4757-841e-be5c630ef424") : secret "dns-default-metrics-tls" not found Jul 21 00:19:38.291880 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:19:38.291801 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:20:22.206231 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:22.206197 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:20:54.713201 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.713168 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-666c4b4fb-psnt5"] Jul 21 00:20:54.715097 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.715081 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:54.718061 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.718037 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 21 00:20:54.718358 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.718341 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 21 00:20:54.718549 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.718535 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hclsz\"" Jul 21 00:20:54.719269 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.719257 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 21 00:20:54.727467 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.727448 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 21 00:20:54.734087 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.734066 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-666c4b4fb-psnt5"] Jul 21 00:20:54.817172 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.817126 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-bound-sa-token\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:54.817367 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.817187 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-ca-trust-extracted\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:54.817367 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.817243 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-certificates\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:54.817367 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.817270 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-image-registry-private-configuration\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:54.817367 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.817304 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f45l\" (UniqueName: \"kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-kube-api-access-4f45l\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:54.817367 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.817329 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-tls\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:54.817367 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.817349 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-installation-pull-secrets\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:54.817581 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.817383 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-trusted-ca\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:54.849362 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.849327 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:20:54.918551 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.918511 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f45l\" (UniqueName: \"kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-kube-api-access-4f45l\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:54.918551 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.918554 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-tls\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:54.918825 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.918579 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-installation-pull-secrets\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:54.918825 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.918628 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-trusted-ca\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:54.918825 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.918664 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-bound-sa-token\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:54.918825 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.918702 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-ca-trust-extracted\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:54.918825 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:20:54.918709 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 00:20:54.918825 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:20:54.918733 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666c4b4fb-psnt5: secret "image-registry-tls" not found Jul 21 00:20:54.918825 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:20:54.918785 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-tls podName:a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79 nodeName:}" failed. No retries permitted until 2026-07-21 00:20:55.418766026 +0000 UTC m=+249.634146335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-tls") pod "image-registry-666c4b4fb-psnt5" (UID: "a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79") : secret "image-registry-tls" not found Jul 21 00:20:54.919314 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.918894 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-certificates\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:54.919314 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.918928 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-image-registry-private-configuration\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:54.919314 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.919118 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-ca-trust-extracted\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:54.919314 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.919310 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-certificates\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:54.919686 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.919666 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-trusted-ca\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:54.921269 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.921244 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-installation-pull-secrets\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:54.921374 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.921268 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-image-registry-private-configuration\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:54.929479 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.929449 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f45l\" (UniqueName: \"kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-kube-api-access-4f45l\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:54.929573 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:54.929521 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-bound-sa-token\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:55.422923 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:55.422870 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-tls\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:55.423088 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:20:55.422991 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 00:20:55.423088 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:20:55.423002 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666c4b4fb-psnt5: secret "image-registry-tls" not found Jul 21 00:20:55.423088 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:20:55.423061 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-tls podName:a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79 nodeName:}" failed. No retries permitted until 2026-07-21 00:20:56.423046356 +0000 UTC m=+250.638426660 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-tls") pod "image-registry-666c4b4fb-psnt5" (UID: "a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79") : secret "image-registry-tls" not found Jul 21 00:20:56.388055 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:56.388024 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:20:56.430574 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:56.430542 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-tls\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:56.430705 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:20:56.430688 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 00:20:56.430705 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:20:56.430702 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666c4b4fb-psnt5: secret "image-registry-tls" not found Jul 21 00:20:56.430823 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:20:56.430747 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-tls podName:a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79 nodeName:}" failed. No retries permitted until 2026-07-21 00:20:58.430733638 +0000 UTC m=+252.646113941 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-tls") pod "image-registry-666c4b4fb-psnt5" (UID: "a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79") : secret "image-registry-tls" not found Jul 21 00:20:57.726754 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:57.726723 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:20:57.784496 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:57.784457 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:20:58.042684 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:58.042589 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs\") pod \"network-metrics-daemon-n8sh6\" (UID: \"982c215c-29b3-45ac-8525-03c61f27e931\") " pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:20:58.042776 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:20:58.042741 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 21 00:20:58.042822 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:20:58.042815 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs podName:982c215c-29b3-45ac-8525-03c61f27e931 nodeName:}" failed. No retries permitted until 2026-07-21 00:23:00.042798792 +0000 UTC m=+374.258179095 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs") pod "network-metrics-daemon-n8sh6" (UID: "982c215c-29b3-45ac-8525-03c61f27e931") : secret "metrics-daemon-secret" not found Jul 21 00:20:58.446093 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:20:58.446044 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-tls\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:20:58.446277 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:20:58.446195 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 00:20:58.446277 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:20:58.446213 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666c4b4fb-psnt5: secret "image-registry-tls" not found Jul 21 00:20:58.446277 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:20:58.446264 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-tls podName:a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79 nodeName:}" failed. No retries permitted until 2026-07-21 00:21:02.446249613 +0000 UTC m=+256.661629916 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-tls") pod "image-registry-666c4b4fb-psnt5" (UID: "a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79") : secret "image-registry-tls" not found Jul 21 00:21:00.172006 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.171972 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:00.267158 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.267121 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:00.364382 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.364346 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:00.422435 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.422346 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:00.474369 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.474339 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:00.619045 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.619013 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-q5w7j"] Jul 21 00:21:00.622087 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.622067 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:00.624873 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.624840 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 21 00:21:00.625697 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.625674 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tx7qk\"" Jul 21 00:21:00.625697 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.625691 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 21 00:21:00.625878 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.625709 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 21 00:21:00.625878 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.625710 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 21 00:21:00.636315 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.636285 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q5w7j"] Jul 21 00:21:00.659004 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.658964 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:00.763578 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.763488 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7182242-ed71-49a7-ac63-fe6ced868d56-crio-socket\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:00.763578 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.763529 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw7cr\" (UniqueName: \"kubernetes.io/projected/a7182242-ed71-49a7-ac63-fe6ced868d56-kube-api-access-tw7cr\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:00.763578 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.763556 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7182242-ed71-49a7-ac63-fe6ced868d56-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:00.763795 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.763675 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7182242-ed71-49a7-ac63-fe6ced868d56-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:00.763795 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.763718 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7182242-ed71-49a7-ac63-fe6ced868d56-data-volume\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:00.864079 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.864042 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7182242-ed71-49a7-ac63-fe6ced868d56-crio-socket\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:00.864079 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.864079 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw7cr\" (UniqueName: \"kubernetes.io/projected/a7182242-ed71-49a7-ac63-fe6ced868d56-kube-api-access-tw7cr\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:00.864315 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.864114 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7182242-ed71-49a7-ac63-fe6ced868d56-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:00.864315 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.864158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7182242-ed71-49a7-ac63-fe6ced868d56-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:00.864315 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.864181 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7182242-ed71-49a7-ac63-fe6ced868d56-crio-socket\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:00.864315 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.864192 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7182242-ed71-49a7-ac63-fe6ced868d56-data-volume\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:00.864315 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:00.864290 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 00:21:00.864315 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:00.864302 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 00:21:00.864562 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:00.864369 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7182242-ed71-49a7-ac63-fe6ced868d56-kube-rbac-proxy-cm podName:a7182242-ed71-49a7-ac63-fe6ced868d56 nodeName:}" failed. No retries permitted until 2026-07-21 00:21:01.364351108 +0000 UTC m=+255.579731412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7182242-ed71-49a7-ac63-fe6ced868d56-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q5w7j" (UID: "a7182242-ed71-49a7-ac63-fe6ced868d56") : configmap "kube-rbac-proxy" not found Jul 21 00:21:00.864562 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:00.864383 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7182242-ed71-49a7-ac63-fe6ced868d56-insights-runtime-extractor-tls podName:a7182242-ed71-49a7-ac63-fe6ced868d56 nodeName:}" failed. No retries permitted until 2026-07-21 00:21:01.364377047 +0000 UTC m=+255.579757350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7182242-ed71-49a7-ac63-fe6ced868d56-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q5w7j" (UID: "a7182242-ed71-49a7-ac63-fe6ced868d56") : secret "insights-runtime-extractor-tls" not found Jul 21 00:21:00.864562 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.864436 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7182242-ed71-49a7-ac63-fe6ced868d56-data-volume\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:00.872175 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.872156 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw7cr\" (UniqueName: \"kubernetes.io/projected/a7182242-ed71-49a7-ac63-fe6ced868d56-kube-api-access-tw7cr\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:00.912208 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:00.912177 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:01.012489 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:01.012443 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:01.368616 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:01.368545 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7182242-ed71-49a7-ac63-fe6ced868d56-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:01.369015 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:01.368673 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 00:21:01.369015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:01.368682 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7182242-ed71-49a7-ac63-fe6ced868d56-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:01.369015 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:01.368744 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7182242-ed71-49a7-ac63-fe6ced868d56-kube-rbac-proxy-cm podName:a7182242-ed71-49a7-ac63-fe6ced868d56 nodeName:}" failed. No retries permitted until 2026-07-21 00:21:02.368721858 +0000 UTC m=+256.584102163 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7182242-ed71-49a7-ac63-fe6ced868d56-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q5w7j" (UID: "a7182242-ed71-49a7-ac63-fe6ced868d56") : configmap "kube-rbac-proxy" not found Jul 21 00:21:01.369015 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:01.368783 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 00:21:01.369015 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:01.368848 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7182242-ed71-49a7-ac63-fe6ced868d56-insights-runtime-extractor-tls podName:a7182242-ed71-49a7-ac63-fe6ced868d56 nodeName:}" failed. No retries permitted until 2026-07-21 00:21:02.368833002 +0000 UTC m=+256.584213308 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7182242-ed71-49a7-ac63-fe6ced868d56-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q5w7j" (UID: "a7182242-ed71-49a7-ac63-fe6ced868d56") : secret "insights-runtime-extractor-tls" not found Jul 21 00:21:02.376976 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:02.376922 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7182242-ed71-49a7-ac63-fe6ced868d56-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:02.376976 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:02.376987 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7182242-ed71-49a7-ac63-fe6ced868d56-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:02.377403 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:02.377084 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 00:21:02.377403 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:02.377088 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 00:21:02.377403 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:02.377138 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7182242-ed71-49a7-ac63-fe6ced868d56-kube-rbac-proxy-cm podName:a7182242-ed71-49a7-ac63-fe6ced868d56 nodeName:}" failed. No retries permitted until 2026-07-21 00:21:04.377123723 +0000 UTC m=+258.592504027 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7182242-ed71-49a7-ac63-fe6ced868d56-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q5w7j" (UID: "a7182242-ed71-49a7-ac63-fe6ced868d56") : configmap "kube-rbac-proxy" not found Jul 21 00:21:02.377403 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:02.377151 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7182242-ed71-49a7-ac63-fe6ced868d56-insights-runtime-extractor-tls podName:a7182242-ed71-49a7-ac63-fe6ced868d56 nodeName:}" failed. No retries permitted until 2026-07-21 00:21:04.377145546 +0000 UTC m=+258.592525851 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7182242-ed71-49a7-ac63-fe6ced868d56-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q5w7j" (UID: "a7182242-ed71-49a7-ac63-fe6ced868d56") : secret "insights-runtime-extractor-tls" not found Jul 21 00:21:02.477792 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:02.477752 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-tls\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:21:02.477966 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:02.477906 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 00:21:02.477966 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:02.477926 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666c4b4fb-psnt5: secret "image-registry-tls" not found Jul 21 00:21:02.478045 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:02.477981 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-tls podName:a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79 nodeName:}" failed. No retries permitted until 2026-07-21 00:21:10.477966389 +0000 UTC m=+264.693346692 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-tls") pod "image-registry-666c4b4fb-psnt5" (UID: "a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79") : secret "image-registry-tls" not found Jul 21 00:21:02.511404 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:02.511366 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:02.749095 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:02.749006 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:03.371003 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:03.370972 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:03.609239 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:03.609212 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-szxbf_583c4be0-d3e2-4a43-b91c-d6a8903569da/dns-node-resolver/0.log" Jul 21 00:21:04.393789 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:04.393754 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7182242-ed71-49a7-ac63-fe6ced868d56-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:04.393977 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:04.393806 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7182242-ed71-49a7-ac63-fe6ced868d56-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:04.393977 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:04.393902 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 00:21:04.393977 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:04.393968 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7182242-ed71-49a7-ac63-fe6ced868d56-kube-rbac-proxy-cm podName:a7182242-ed71-49a7-ac63-fe6ced868d56 nodeName:}" failed. No retries permitted until 2026-07-21 00:21:08.393950518 +0000 UTC m=+262.609330822 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7182242-ed71-49a7-ac63-fe6ced868d56-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q5w7j" (UID: "a7182242-ed71-49a7-ac63-fe6ced868d56") : configmap "kube-rbac-proxy" not found Jul 21 00:21:04.394083 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:04.393902 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 00:21:04.394083 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:04.394052 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7182242-ed71-49a7-ac63-fe6ced868d56-insights-runtime-extractor-tls podName:a7182242-ed71-49a7-ac63-fe6ced868d56 nodeName:}" failed. No retries permitted until 2026-07-21 00:21:08.394036425 +0000 UTC m=+262.609416728 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7182242-ed71-49a7-ac63-fe6ced868d56-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q5w7j" (UID: "a7182242-ed71-49a7-ac63-fe6ced868d56") : secret "insights-runtime-extractor-tls" not found Jul 21 00:21:04.420003 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:04.419971 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:04.610024 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:04.609996 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p76tl_2bd25715-f83d-42fa-953e-ba07d8b7da50/node-ca/0.log" Jul 21 00:21:04.654326 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:04.654240 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:04.750390 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:04.750356 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:04.906027 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:04.905945 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:05.022782 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:05.022748 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:05.127843 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:05.127814 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:05.208836 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:05.208764 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-br7vh_5046e73d-17d6-4282-9387-c665c60f7ca9/serve-healthcheck-canary/0.log" Jul 21 00:21:05.912537 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:05.912475 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:06.014077 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:06.014040 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:06.319453 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:06.319416 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:06.718107 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:06.718015 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:07.117098 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:07.117051 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:07.518734 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:07.518642 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:07.721727 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:07.721698 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:08.119551 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:08.119515 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:08.323678 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:08.323633 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:08.425542 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:08.425456 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7182242-ed71-49a7-ac63-fe6ced868d56-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:08.425542 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:08.425521 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7182242-ed71-49a7-ac63-fe6ced868d56-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:08.425742 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:08.425657 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 00:21:08.425742 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:08.425658 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 00:21:08.425742 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:08.425715 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7182242-ed71-49a7-ac63-fe6ced868d56-kube-rbac-proxy-cm podName:a7182242-ed71-49a7-ac63-fe6ced868d56 nodeName:}" failed. No retries permitted until 2026-07-21 00:21:16.425700219 +0000 UTC m=+270.641080522 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7182242-ed71-49a7-ac63-fe6ced868d56-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q5w7j" (UID: "a7182242-ed71-49a7-ac63-fe6ced868d56") : configmap "kube-rbac-proxy" not found Jul 21 00:21:08.425742 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:08.425728 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7182242-ed71-49a7-ac63-fe6ced868d56-insights-runtime-extractor-tls podName:a7182242-ed71-49a7-ac63-fe6ced868d56 nodeName:}" failed. No retries permitted until 2026-07-21 00:21:16.425722095 +0000 UTC m=+270.641102398 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7182242-ed71-49a7-ac63-fe6ced868d56-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q5w7j" (UID: "a7182242-ed71-49a7-ac63-fe6ced868d56") : secret "insights-runtime-extractor-tls" not found Jul 21 00:21:08.540587 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:08.540553 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:08.720892 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:08.720793 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:09.313503 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:09.313460 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:09.515911 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:09.515870 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:09.718811 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:09.718710 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:09.928074 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:09.928042 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:10.512827 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:10.512792 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:10.539872 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:10.539818 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-tls\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:21:10.542313 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:10.542295 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-tls\") pod \"image-registry-666c4b4fb-psnt5\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:21:10.624648 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:10.624589 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:21:10.751712 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:10.751678 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-666c4b4fb-psnt5"] Jul 21 00:21:10.754633 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:21:10.754589 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d0eb64_f2ae_4ecb_9d8d_744c8bd24d79.slice/crio-a1606e1bb5d40934303d129248088e9ebdae66299f730da84aff41915997c880 WatchSource:0}: Error finding container a1606e1bb5d40934303d129248088e9ebdae66299f730da84aff41915997c880: Status 404 returned error can't find the container with id a1606e1bb5d40934303d129248088e9ebdae66299f730da84aff41915997c880 Jul 21 00:21:10.916870 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:10.916839 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:10.974219 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:10.974182 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:11.083387 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:11.083352 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" event={"ID":"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79","Type":"ContainerStarted","Data":"c559e78ea5bd88c395615c4e1516f2778608b934be8fd9793414d35166503883"} Jul 21 00:21:11.083559 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:11.083400 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" event={"ID":"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79","Type":"ContainerStarted","Data":"a1606e1bb5d40934303d129248088e9ebdae66299f730da84aff41915997c880"} Jul 21 00:21:11.083559 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:11.083430 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:21:11.103028 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:11.102978 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" podStartSLOduration=17.102961788 podStartE2EDuration="17.102961788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:21:11.101442324 +0000 UTC m=+265.316822650" watchObservedRunningTime="2026-07-21 00:21:11.102961788 +0000 UTC m=+265.318342112" Jul 21 00:21:11.315413 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:11.315385 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:11.516688 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:11.516579 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:11.719094 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:11.719059 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:11.912985 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:11.912947 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:12.890763 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:12.890706 2580 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" podUID="2a83348f-884c-40d7-b24a-77deeadc39ff" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 21 00:21:12.972179 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:12.972145 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:13.122403 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:13.122368 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:13.419352 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:13.419318 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:13.520706 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:13.520677 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:13.717366 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:13.717247 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:14.121497 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:14.121464 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:14.667854 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:14.667816 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:14.768866 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:14.768831 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:14.935265 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:14.935173 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:15.125134 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:15.125101 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:16.486544 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:16.486510 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7182242-ed71-49a7-ac63-fe6ced868d56-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:16.486544 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:16.486552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7182242-ed71-49a7-ac63-fe6ced868d56-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:16.487053 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:16.487034 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7182242-ed71-49a7-ac63-fe6ced868d56-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:16.488912 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:16.488895 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7182242-ed71-49a7-ac63-fe6ced868d56-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q5w7j\" (UID: \"a7182242-ed71-49a7-ac63-fe6ced868d56\") " pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:16.531249 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:16.531206 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q5w7j" Jul 21 00:21:16.660275 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:16.660244 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q5w7j"] Jul 21 00:21:16.663405 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:21:16.663379 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7182242_ed71_49a7_ac63_fe6ced868d56.slice/crio-799aad5501d4355b06d0955ed696fdce6bba29b3691b240765d3f2b4a237e2f3 WatchSource:0}: Error finding container 799aad5501d4355b06d0955ed696fdce6bba29b3691b240765d3f2b4a237e2f3: Status 404 returned error can't find the container with id 799aad5501d4355b06d0955ed696fdce6bba29b3691b240765d3f2b4a237e2f3 Jul 21 00:21:17.102372 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:17.102336 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q5w7j" event={"ID":"a7182242-ed71-49a7-ac63-fe6ced868d56","Type":"ContainerStarted","Data":"799aad5501d4355b06d0955ed696fdce6bba29b3691b240765d3f2b4a237e2f3"} Jul 21 00:21:18.056973 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:18.056934 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:21.074003 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:21.073956 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-666c4b4fb-psnt5"] Jul 21 00:21:21.179103 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:21.179055 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:21.273632 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:21.273556 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:22.121987 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:22.121948 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q5w7j" event={"ID":"a7182242-ed71-49a7-ac63-fe6ced868d56","Type":"ContainerStarted","Data":"bdfbd70d463685c92442cd3683b69182bd458dc0075acfaf394e077022022417"} Jul 21 00:21:22.195414 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:22.195370 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:22.295448 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:22.295401 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:22.373057 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:22.372931 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:22.725884 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:22.725787 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:22.888305 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:22.888261 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:22.890274 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:22.890232 2580 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" podUID="2a83348f-884c-40d7-b24a-77deeadc39ff" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 21 00:21:22.994545 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:22.994466 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:23.111787 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:23.111680 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:23.211378 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:23.211280 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:23.317436 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:23.317396 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:23.420587 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:23.420553 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:23.518685 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:23.518637 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:23.581497 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:23.581409 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:23.675339 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:23.675303 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:24.066306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:24.066219 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:24.263272 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:24.263233 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:24.585949 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:24.585910 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:25.090249 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:25.090214 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:25.132322 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:25.132240 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q5w7j" event={"ID":"a7182242-ed71-49a7-ac63-fe6ced868d56","Type":"ContainerStarted","Data":"b60c10915f657444897d571402fcc397d5cce4e562f4e2177aad63344f35fe41"} Jul 21 00:21:25.191899 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:25.191862 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:28.792234 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:28.792184 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pz4d4" podUID="59620b96-9827-4757-841e-be5c630ef424" Jul 21 00:21:29.146948 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:29.146911 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q5w7j" event={"ID":"a7182242-ed71-49a7-ac63-fe6ced868d56","Type":"ContainerStarted","Data":"4e0d39ca3b0e840846ee3d621847d4656d648578cd6f445e3d85af23300703c5"} Jul 21 00:21:29.146948 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:29.146930 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pz4d4" Jul 21 00:21:29.164855 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:29.164779 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-q5w7j" podStartSLOduration=17.089107383 podStartE2EDuration="29.164763541s" totalImagesPullingTime="12.075656158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 00:21:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:21:29.164037764 +0000 UTC m=+283.379418089" watchObservedRunningTime="2026-07-21 00:21:29.164763541 +0000 UTC m=+283.380143900" Jul 21 00:21:30.486019 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:30.485976 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:30.588317 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:30.588289 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:30.653490 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:30.653423 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-j7bpq"] Jul 21 00:21:30.655651 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:30.655629 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-j7bpq" Jul 21 00:21:30.658066 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:30.658044 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-pl5tp\"" Jul 21 00:21:30.658198 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:30.658043 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 21 00:21:30.665116 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:30.665090 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-j7bpq"] Jul 21 00:21:30.685011 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:30.684974 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:30.696082 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:30.696048 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2beddce2-7b22-48ef-9399-df9c9c0f9be4-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-j7bpq\" (UID: \"2beddce2-7b22-48ef-9399-df9c9c0f9be4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-j7bpq" Jul 21 00:21:30.796443 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:30.796351 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2beddce2-7b22-48ef-9399-df9c9c0f9be4-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-j7bpq\" (UID: \"2beddce2-7b22-48ef-9399-df9c9c0f9be4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-j7bpq" Jul 21 00:21:30.796644 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:30.796536 2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 21 00:21:30.796715 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:30.796652 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2beddce2-7b22-48ef-9399-df9c9c0f9be4-tls-certificates podName:2beddce2-7b22-48ef-9399-df9c9c0f9be4 nodeName:}" failed. No retries permitted until 2026-07-21 00:21:31.296592228 +0000 UTC m=+285.511972539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/2beddce2-7b22-48ef-9399-df9c9c0f9be4-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-j7bpq" (UID: "2beddce2-7b22-48ef-9399-df9c9c0f9be4") : secret "prometheus-operator-admission-webhook-tls" not found Jul 21 00:21:30.944784 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:30.944750 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:31.045885 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:31.045848 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:31.079744 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:31.079711 2580 patch_prober.go:28] interesting pod/image-registry-666c4b4fb-psnt5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 21 00:21:31.079936 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:31.079781 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" podUID="a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 00:21:31.146714 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:31.146673 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:31.301449 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:31.301415 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2beddce2-7b22-48ef-9399-df9c9c0f9be4-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-j7bpq\" (UID: \"2beddce2-7b22-48ef-9399-df9c9c0f9be4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-j7bpq" Jul 21 00:21:31.303893 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:31.303863 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2beddce2-7b22-48ef-9399-df9c9c0f9be4-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-j7bpq\" (UID: \"2beddce2-7b22-48ef-9399-df9c9c0f9be4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-j7bpq" Jul 21 00:21:31.563976 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:31.563942 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-j7bpq" Jul 21 00:21:31.690968 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:31.690938 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-j7bpq"] Jul 21 00:21:31.693778 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:21:31.693736 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2beddce2_7b22_48ef_9399_df9c9c0f9be4.slice/crio-1977507b4f8dab528b809693db2fea35091b7d086079040c3f32576e3f4cf4c0 WatchSource:0}: Error finding container 1977507b4f8dab528b809693db2fea35091b7d086079040c3f32576e3f4cf4c0: Status 404 returned error can't find the container with id 1977507b4f8dab528b809693db2fea35091b7d086079040c3f32576e3f4cf4c0 Jul 21 00:21:32.106703 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:32.106662 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls\") pod \"dns-default-pz4d4\" (UID: \"59620b96-9827-4757-841e-be5c630ef424\") " pod="openshift-dns/dns-default-pz4d4" Jul 21 00:21:32.109198 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:32.109171 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59620b96-9827-4757-841e-be5c630ef424-metrics-tls\") pod \"dns-default-pz4d4\" (UID: \"59620b96-9827-4757-841e-be5c630ef424\") " pod="openshift-dns/dns-default-pz4d4" Jul 21 00:21:32.151772 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:32.151742 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-26lqt\"" Jul 21 00:21:32.157064 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:32.157035 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-j7bpq" event={"ID":"2beddce2-7b22-48ef-9399-df9c9c0f9be4","Type":"ContainerStarted","Data":"1977507b4f8dab528b809693db2fea35091b7d086079040c3f32576e3f4cf4c0"} Jul 21 00:21:32.159215 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:32.159198 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pz4d4" Jul 21 00:21:32.295955 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:32.295889 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pz4d4"] Jul 21 00:21:32.298830 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:21:32.298797 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59620b96_9827_4757_841e_be5c630ef424.slice/crio-b4aed7136673f3e8529c05ff5a69103a0bb6920df001ac16334cc980b48b2170 WatchSource:0}: Error finding container b4aed7136673f3e8529c05ff5a69103a0bb6920df001ac16334cc980b48b2170: Status 404 returned error can't find the container with id b4aed7136673f3e8529c05ff5a69103a0bb6920df001ac16334cc980b48b2170 Jul 21 00:21:32.890845 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:32.890789 2580 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" podUID="2a83348f-884c-40d7-b24a-77deeadc39ff" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 21 00:21:32.891305 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:32.890871 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" Jul 21 00:21:32.891494 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:32.891454 2580 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"fc10e821b61177ba5daeca5d96203a023813164846afa6487b42b1b9ee573ad2"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 21 00:21:32.891558 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:32.891526 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" podUID="2a83348f-884c-40d7-b24a-77deeadc39ff" containerName="service-proxy" containerID="cri-o://fc10e821b61177ba5daeca5d96203a023813164846afa6487b42b1b9ee573ad2" gracePeriod=30 Jul 21 00:21:33.162339 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:33.162246 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-j7bpq" event={"ID":"2beddce2-7b22-48ef-9399-df9c9c0f9be4","Type":"ContainerStarted","Data":"946446f9489a029d5e9cd08cca850a8bebaa181328441a55b2ad86e4260a69e0"} Jul 21 00:21:33.162591 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:33.162537 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-j7bpq" Jul 21 00:21:33.163724 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:33.163691 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pz4d4" event={"ID":"59620b96-9827-4757-841e-be5c630ef424","Type":"ContainerStarted","Data":"b4aed7136673f3e8529c05ff5a69103a0bb6920df001ac16334cc980b48b2170"} Jul 21 00:21:33.166919 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:33.166888 2580 generic.go:356] "Generic (PLEG): container finished" podID="2a83348f-884c-40d7-b24a-77deeadc39ff" containerID="fc10e821b61177ba5daeca5d96203a023813164846afa6487b42b1b9ee573ad2" exitCode=2 Jul 21 00:21:33.167035 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:33.166966 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" event={"ID":"2a83348f-884c-40d7-b24a-77deeadc39ff","Type":"ContainerDied","Data":"fc10e821b61177ba5daeca5d96203a023813164846afa6487b42b1b9ee573ad2"} Jul 21 00:21:33.167035 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:33.167002 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-546885c98-xkf2n" event={"ID":"2a83348f-884c-40d7-b24a-77deeadc39ff","Type":"ContainerStarted","Data":"6a636a647079c44937b4e976a9d4d086c3b4011ee116618e59f3064474b26ec9"} Jul 21 00:21:33.168115 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:33.168093 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-j7bpq" Jul 21 00:21:33.191837 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:33.191783 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-j7bpq" podStartSLOduration=1.8513148529999999 podStartE2EDuration="3.191768017s" totalImagesPullingTime="1.340453164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:21:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:21:33.190501152 +0000 UTC m=+287.405881478" watchObservedRunningTime="2026-07-21 00:21:33.191768017 +0000 UTC m=+287.407148341" Jul 21 00:21:33.687421 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:33.687386 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:33.772973 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:33.772938 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:33.829785 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:33.829677 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:34.173728 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:34.173694 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pz4d4" event={"ID":"59620b96-9827-4757-841e-be5c630ef424","Type":"ContainerStarted","Data":"784f8db553a930253dd0db3a748b8a30c0faa997d5bb0e17cc882386dce60661"} Jul 21 00:21:34.174128 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:34.173749 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pz4d4" event={"ID":"59620b96-9827-4757-841e-be5c630ef424","Type":"ContainerStarted","Data":"cd12174c82fabee61962c870cee6ace4bb4f716431dfc54a38dd376b488e12d4"} Jul 21 00:21:34.174128 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:34.173917 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pz4d4" Jul 21 00:21:34.192433 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:34.192329 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pz4d4" podStartSLOduration=250.553710026 podStartE2EDuration="4m12.19231263s" totalImagesPullingTime="1.638602604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:21:34.191924608 +0000 UTC m=+288.407304945" watchObservedRunningTime="2026-07-21 00:21:34.19231263 +0000 UTC m=+288.407692968" Jul 21 00:21:36.012652 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.012619 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:36.490322 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.490290 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85c6d96c66-rq4z4"] Jul 21 00:21:36.492495 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.492468 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:21:36.496440 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.495492 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 21 00:21:36.497498 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.496774 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 21 00:21:36.497498 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.497006 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 21 00:21:36.497498 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.497213 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 21 00:21:36.497498 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.497404 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ks6p9\"" Jul 21 00:21:36.497882 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.497715 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 21 00:21:36.497882 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.497752 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 21 00:21:36.498114 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.498090 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 21 00:21:36.504090 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.504068 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85c6d96c66-rq4z4"] Jul 21 00:21:36.542677 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.542642 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xrrn\" (UniqueName: \"kubernetes.io/projected/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-kube-api-access-8xrrn\") pod \"console-85c6d96c66-rq4z4\" (UID: \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\") " pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:21:36.542853 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.542688 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-console-serving-cert\") pod \"console-85c6d96c66-rq4z4\" (UID: \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\") " pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:21:36.542853 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.542726 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-console-oauth-config\") pod \"console-85c6d96c66-rq4z4\" (UID: \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\") " pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:21:36.542853 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.542818 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-service-ca\") pod \"console-85c6d96c66-rq4z4\" (UID: \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\") " pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:21:36.543020 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.542912 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-oauth-serving-cert\") pod \"console-85c6d96c66-rq4z4\" (UID: \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\") " pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:21:36.543020 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.542964 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-console-config\") pod \"console-85c6d96c66-rq4z4\" (UID: \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\") " pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:21:36.643624 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.643565 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-oauth-serving-cert\") pod \"console-85c6d96c66-rq4z4\" (UID: \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\") " pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:21:36.643838 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.643636 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-console-config\") pod \"console-85c6d96c66-rq4z4\" (UID: \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\") " pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:21:36.643838 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.643664 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xrrn\" (UniqueName: \"kubernetes.io/projected/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-kube-api-access-8xrrn\") pod \"console-85c6d96c66-rq4z4\" (UID: \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\") " pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:21:36.643838 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.643697 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-console-serving-cert\") pod \"console-85c6d96c66-rq4z4\" (UID: \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\") " pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:21:36.643838 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.643752 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-console-oauth-config\") pod \"console-85c6d96c66-rq4z4\" (UID: \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\") " pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:21:36.643838 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.643775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-service-ca\") pod \"console-85c6d96c66-rq4z4\" (UID: \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\") " pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:21:36.644404 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.644378 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-oauth-serving-cert\") pod \"console-85c6d96c66-rq4z4\" (UID: \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\") " pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:21:36.644522 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.644458 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-service-ca\") pod \"console-85c6d96c66-rq4z4\" (UID: \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\") " pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:21:36.644522 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.644507 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-console-config\") pod \"console-85c6d96c66-rq4z4\" (UID: \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\") " pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:21:36.646324 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.646301 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-console-serving-cert\") pod \"console-85c6d96c66-rq4z4\" (UID: \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\") " pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:21:36.646408 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.646389 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-console-oauth-config\") pod \"console-85c6d96c66-rq4z4\" (UID: \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\") " pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:21:36.652339 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.652317 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xrrn\" (UniqueName: \"kubernetes.io/projected/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-kube-api-access-8xrrn\") pod \"console-85c6d96c66-rq4z4\" (UID: \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\") " pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:21:36.804264 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.804229 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:21:36.936793 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:36.936760 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85c6d96c66-rq4z4"] Jul 21 00:21:36.939566 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:21:36.939532 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e5fc8a_8a6e_4627_adcf_953ad08a11d1.slice/crio-04dfa5de582635a670513f07ffa7940c437375ef31944d8514476d4d3c340bd6 WatchSource:0}: Error finding container 04dfa5de582635a670513f07ffa7940c437375ef31944d8514476d4d3c340bd6: Status 404 returned error can't find the container with id 04dfa5de582635a670513f07ffa7940c437375ef31944d8514476d4d3c340bd6 Jul 21 00:21:37.184716 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:37.184632 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c6d96c66-rq4z4" event={"ID":"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1","Type":"ContainerStarted","Data":"04dfa5de582635a670513f07ffa7940c437375ef31944d8514476d4d3c340bd6"} Jul 21 00:21:38.695983 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:38.695942 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:38.757700 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:38.757643 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:38.852413 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:38.852367 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:38.943723 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:38.943688 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:39.069546 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.069492 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:39.100403 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.100366 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-f9hxl"] Jul 21 00:21:39.103121 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.103095 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.105995 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.105937 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 21 00:21:39.105995 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.105938 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 21 00:21:39.106255 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.106235 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zvvjz\"" Jul 21 00:21:39.106371 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.106354 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 21 00:21:39.106957 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.106931 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 21 00:21:39.107056 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.106970 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 21 00:21:39.107056 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.106940 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 21 00:21:39.163724 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.163689 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bb7d385b-b316-481f-9cad-354c798728cc-root\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.163724 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.163734 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bb7d385b-b316-481f-9cad-354c798728cc-node-exporter-textfile\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.163993 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.163758 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bb7d385b-b316-481f-9cad-354c798728cc-node-exporter-tls\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.163993 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.163785 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb7d385b-b316-481f-9cad-354c798728cc-sys\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.163993 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.163869 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bb7d385b-b316-481f-9cad-354c798728cc-node-exporter-wtmp\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.163993 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.163902 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb7d385b-b316-481f-9cad-354c798728cc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.163993 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.163930 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94rsf\" (UniqueName: \"kubernetes.io/projected/bb7d385b-b316-481f-9cad-354c798728cc-kube-api-access-94rsf\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.163993 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.163992 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb7d385b-b316-481f-9cad-354c798728cc-metrics-client-ca\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.164289 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.164024 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bb7d385b-b316-481f-9cad-354c798728cc-node-exporter-accelerators-collector-config\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.196486 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.196429 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:39.265420 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.265380 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb7d385b-b316-481f-9cad-354c798728cc-metrics-client-ca\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.265652 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.265441 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bb7d385b-b316-481f-9cad-354c798728cc-node-exporter-accelerators-collector-config\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.265652 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.265481 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bb7d385b-b316-481f-9cad-354c798728cc-root\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.265652 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.265508 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bb7d385b-b316-481f-9cad-354c798728cc-node-exporter-textfile\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.265652 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.265532 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bb7d385b-b316-481f-9cad-354c798728cc-node-exporter-tls\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.265652 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.265562 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb7d385b-b316-481f-9cad-354c798728cc-sys\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.265652 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.265627 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bb7d385b-b316-481f-9cad-354c798728cc-root\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.265652 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.265651 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bb7d385b-b316-481f-9cad-354c798728cc-node-exporter-wtmp\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.266007 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.265693 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb7d385b-b316-481f-9cad-354c798728cc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.266007 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.265730 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94rsf\" (UniqueName: \"kubernetes.io/projected/bb7d385b-b316-481f-9cad-354c798728cc-kube-api-access-94rsf\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.266007 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.265784 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bb7d385b-b316-481f-9cad-354c798728cc-node-exporter-wtmp\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.266007 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.265837 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb7d385b-b316-481f-9cad-354c798728cc-sys\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.266199 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.266169 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb7d385b-b316-481f-9cad-354c798728cc-metrics-client-ca\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.266416 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.266392 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bb7d385b-b316-481f-9cad-354c798728cc-node-exporter-textfile\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.266668 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.266619 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bb7d385b-b316-481f-9cad-354c798728cc-node-exporter-accelerators-collector-config\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.268567 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.268543 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb7d385b-b316-481f-9cad-354c798728cc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.268796 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.268766 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bb7d385b-b316-481f-9cad-354c798728cc-node-exporter-tls\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.274463 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.274435 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94rsf\" (UniqueName: \"kubernetes.io/projected/bb7d385b-b316-481f-9cad-354c798728cc-kube-api-access-94rsf\") pod \"node-exporter-f9hxl\" (UID: \"bb7d385b-b316-481f-9cad-354c798728cc\") " pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.278815 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.278733 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:39.380636 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.380523 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:39.414723 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.414677 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f9hxl" Jul 21 00:21:39.480862 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.480827 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:39.669724 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.669635 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:39.770503 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:39.770463 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:40.090020 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:40.089978 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:40.123952 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:21:40.123911 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7d385b_b316_481f_9cad_354c798728cc.slice/crio-5314fa47298eebef636c5450354c6d46a086be24a4d39bd5eaa5f94125e22c8a WatchSource:0}: Error finding container 5314fa47298eebef636c5450354c6d46a086be24a4d39bd5eaa5f94125e22c8a: Status 404 returned error can't find the container with id 5314fa47298eebef636c5450354c6d46a086be24a4d39bd5eaa5f94125e22c8a Jul 21 00:21:40.165081 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:40.165053 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:40.196934 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:40.196896 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f9hxl" event={"ID":"bb7d385b-b316-481f-9cad-354c798728cc","Type":"ContainerStarted","Data":"5314fa47298eebef636c5450354c6d46a086be24a4d39bd5eaa5f94125e22c8a"} Jul 21 00:21:40.396249 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:40.396147 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:41.079685 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:41.079636 2580 patch_prober.go:28] interesting pod/image-registry-666c4b4fb-psnt5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 21 00:21:41.080211 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:41.079710 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" podUID="a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 00:21:41.202450 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:41.202417 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c6d96c66-rq4z4" event={"ID":"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1","Type":"ContainerStarted","Data":"21a01b092f4d659cf233a17e53ed6c31553028a48ddc6004920d8936c2c4f3c6"} Jul 21 00:21:41.218727 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:41.218663 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85c6d96c66-rq4z4" podStartSLOduration=2.008015148 podStartE2EDuration="5.218641486s" totalImagesPullingTime="3.210626338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:21:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:21:41.217416343 +0000 UTC m=+295.432796726" watchObservedRunningTime="2026-07-21 00:21:41.218641486 +0000 UTC m=+295.434021815" Jul 21 00:21:41.292910 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:21:41.292869 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-n8sh6" podUID="982c215c-29b3-45ac-8525-03c61f27e931" Jul 21 00:21:41.885920 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:41.885882 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:42.006036 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.005998 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:42.083754 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.083722 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:42.088138 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.088109 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-59b7fcc89f-frkvp"] Jul 21 00:21:42.091388 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.091366 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.093754 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.093734 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 21 00:21:42.093928 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.093886 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 21 00:21:42.094043 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.094019 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 21 00:21:42.094043 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.094023 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 21 00:21:42.094242 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.094090 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-kt7p9\"" Jul 21 00:21:42.095008 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.094968 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7gtpp3sljmia1\"" Jul 21 00:21:42.095008 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.094986 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 21 00:21:42.102649 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.102623 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-59b7fcc89f-frkvp"] Jul 21 00:21:42.162156 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.162061 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:42.187084 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.187049 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9783a8f-d1a0-4d04-8452-e8cc6520cf5e-secret-grpc-tls\") pod \"thanos-querier-59b7fcc89f-frkvp\" (UID: \"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e\") " pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.187304 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.187112 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm4ht\" (UniqueName: \"kubernetes.io/projected/d9783a8f-d1a0-4d04-8452-e8cc6520cf5e-kube-api-access-bm4ht\") pod \"thanos-querier-59b7fcc89f-frkvp\" (UID: \"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e\") " pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.187304 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.187142 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9783a8f-d1a0-4d04-8452-e8cc6520cf5e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59b7fcc89f-frkvp\" (UID: \"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e\") " pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.187304 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.187174 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9783a8f-d1a0-4d04-8452-e8cc6520cf5e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59b7fcc89f-frkvp\" (UID: \"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e\") " pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.187304 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.187246 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d9783a8f-d1a0-4d04-8452-e8cc6520cf5e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59b7fcc89f-frkvp\" (UID: \"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e\") " pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.187304 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.187272 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9783a8f-d1a0-4d04-8452-e8cc6520cf5e-metrics-client-ca\") pod \"thanos-querier-59b7fcc89f-frkvp\" (UID: \"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e\") " pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.187478 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.187316 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d9783a8f-d1a0-4d04-8452-e8cc6520cf5e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59b7fcc89f-frkvp\" (UID: \"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e\") " pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.187478 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.187362 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d9783a8f-d1a0-4d04-8452-e8cc6520cf5e-secret-thanos-querier-tls\") pod \"thanos-querier-59b7fcc89f-frkvp\" (UID: \"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e\") " pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.207358 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.207328 2580 generic.go:356] "Generic (PLEG): container finished" podID="bb7d385b-b316-481f-9cad-354c798728cc" containerID="b5fc250fc011985833389ae02f82cb3961284b6679643a7efdd6294effedffeb" exitCode=0 Jul 21 00:21:42.207480 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.207422 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f9hxl" event={"ID":"bb7d385b-b316-481f-9cad-354c798728cc","Type":"ContainerDied","Data":"b5fc250fc011985833389ae02f82cb3961284b6679643a7efdd6294effedffeb"} Jul 21 00:21:42.287893 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.287866 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d9783a8f-d1a0-4d04-8452-e8cc6520cf5e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59b7fcc89f-frkvp\" (UID: \"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e\") " pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.288017 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.287921 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9783a8f-d1a0-4d04-8452-e8cc6520cf5e-metrics-client-ca\") pod \"thanos-querier-59b7fcc89f-frkvp\" (UID: \"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e\") " pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.288265 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.288246 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d9783a8f-d1a0-4d04-8452-e8cc6520cf5e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59b7fcc89f-frkvp\" (UID: \"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e\") " pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.288364 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.288344 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d9783a8f-d1a0-4d04-8452-e8cc6520cf5e-secret-thanos-querier-tls\") pod \"thanos-querier-59b7fcc89f-frkvp\" (UID: \"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e\") " pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.288436 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.288409 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9783a8f-d1a0-4d04-8452-e8cc6520cf5e-secret-grpc-tls\") pod \"thanos-querier-59b7fcc89f-frkvp\" (UID: \"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e\") " pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.288570 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.288549 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm4ht\" (UniqueName: \"kubernetes.io/projected/d9783a8f-d1a0-4d04-8452-e8cc6520cf5e-kube-api-access-bm4ht\") pod \"thanos-querier-59b7fcc89f-frkvp\" (UID: \"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e\") " pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.288678 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.288626 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9783a8f-d1a0-4d04-8452-e8cc6520cf5e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59b7fcc89f-frkvp\" (UID: \"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e\") " pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.288678 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.288656 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9783a8f-d1a0-4d04-8452-e8cc6520cf5e-metrics-client-ca\") pod \"thanos-querier-59b7fcc89f-frkvp\" (UID: \"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e\") " pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.288775 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.288700 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9783a8f-d1a0-4d04-8452-e8cc6520cf5e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59b7fcc89f-frkvp\" (UID: \"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e\") " pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.291466 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.291418 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d9783a8f-d1a0-4d04-8452-e8cc6520cf5e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59b7fcc89f-frkvp\" (UID: \"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e\") " pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.292032 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.292002 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d9783a8f-d1a0-4d04-8452-e8cc6520cf5e-secret-thanos-querier-tls\") pod \"thanos-querier-59b7fcc89f-frkvp\" (UID: \"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e\") " pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.292318 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.292290 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d9783a8f-d1a0-4d04-8452-e8cc6520cf5e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59b7fcc89f-frkvp\" (UID: \"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e\") " pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.293010 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.292990 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9783a8f-d1a0-4d04-8452-e8cc6520cf5e-secret-grpc-tls\") pod \"thanos-querier-59b7fcc89f-frkvp\" (UID: \"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e\") " pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.293085 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.293069 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9783a8f-d1a0-4d04-8452-e8cc6520cf5e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59b7fcc89f-frkvp\" (UID: \"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e\") " pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.293459 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.293438 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9783a8f-d1a0-4d04-8452-e8cc6520cf5e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59b7fcc89f-frkvp\" (UID: \"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e\") " pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.297001 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.296976 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm4ht\" (UniqueName: \"kubernetes.io/projected/d9783a8f-d1a0-4d04-8452-e8cc6520cf5e-kube-api-access-bm4ht\") pod \"thanos-querier-59b7fcc89f-frkvp\" (UID: \"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e\") " pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.400265 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.400227 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:42.554992 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.554960 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-59b7fcc89f-frkvp"] Jul 21 00:21:42.557670 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:21:42.557645 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9783a8f_d1a0_4d04_8452_e8cc6520cf5e.slice/crio-6d60ac8f5da6be4dc326e7265198f7775ed6e5d940f2dd000ea53f3ad26e589f WatchSource:0}: Error finding container 6d60ac8f5da6be4dc326e7265198f7775ed6e5d940f2dd000ea53f3ad26e589f: Status 404 returned error can't find the container with id 6d60ac8f5da6be4dc326e7265198f7775ed6e5d940f2dd000ea53f3ad26e589f Jul 21 00:21:42.680571 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:42.680512 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:43.091630 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:43.091582 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:43.142338 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:43.142300 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:43.213401 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:43.213364 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" event={"ID":"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e","Type":"ContainerStarted","Data":"6d60ac8f5da6be4dc326e7265198f7775ed6e5d940f2dd000ea53f3ad26e589f"} Jul 21 00:21:43.215706 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:43.215672 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f9hxl" event={"ID":"bb7d385b-b316-481f-9cad-354c798728cc","Type":"ContainerStarted","Data":"071d6293547527dc664cc69d5a69b765f286f544743375152bc2d4d275159268"} Jul 21 00:21:43.215865 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:43.215730 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f9hxl" event={"ID":"bb7d385b-b316-481f-9cad-354c798728cc","Type":"ContainerStarted","Data":"62bdeca1443c64a9f04a3719629e1ffe9471ccd8bc70fe4ad22b77cd70e0ab94"} Jul 21 00:21:43.236663 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:43.236591 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-f9hxl" podStartSLOduration=3.108263012 podStartE2EDuration="4.23657322s" totalImagesPullingTime="1.052890035s" totalInitContainerRuntime="75.420173ms" isStatefulPod=true podCreationTimestamp="2026-07-21 00:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:21:43.235986417 +0000 UTC m=+297.451366744" watchObservedRunningTime="2026-07-21 00:21:43.23657322 +0000 UTC m=+297.451953594" Jul 21 00:21:43.859235 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:43.859178 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:43.867108 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:43.867078 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ccs8p"] Jul 21 00:21:43.870428 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:43.870402 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ccs8p" Jul 21 00:21:43.872789 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:43.872765 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6mtxw\"" Jul 21 00:21:43.872918 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:43.872767 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 21 00:21:43.877174 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:43.877148 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ccs8p"] Jul 21 00:21:43.955469 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:43.955414 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:44.003201 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:44.003157 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/13b254c8-1a1e-40db-b169-9c13a98430f5-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-ccs8p\" (UID: \"13b254c8-1a1e-40db-b169-9c13a98430f5\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ccs8p" Jul 21 00:21:44.003201 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:44.003202 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smngq\" (UniqueName: \"kubernetes.io/projected/13b254c8-1a1e-40db-b169-9c13a98430f5-kube-api-access-smngq\") pod \"monitoring-plugin-7b7b7bc8bf-ccs8p\" (UID: \"13b254c8-1a1e-40db-b169-9c13a98430f5\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ccs8p" Jul 21 00:21:44.104684 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:44.104630 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/13b254c8-1a1e-40db-b169-9c13a98430f5-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-ccs8p\" (UID: \"13b254c8-1a1e-40db-b169-9c13a98430f5\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ccs8p" Jul 21 00:21:44.104684 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:44.104690 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smngq\" (UniqueName: \"kubernetes.io/projected/13b254c8-1a1e-40db-b169-9c13a98430f5-kube-api-access-smngq\") pod \"monitoring-plugin-7b7b7bc8bf-ccs8p\" (UID: \"13b254c8-1a1e-40db-b169-9c13a98430f5\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ccs8p" Jul 21 00:21:44.107517 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:44.107484 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/13b254c8-1a1e-40db-b169-9c13a98430f5-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-ccs8p\" (UID: \"13b254c8-1a1e-40db-b169-9c13a98430f5\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ccs8p" Jul 21 00:21:44.113197 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:44.113135 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smngq\" (UniqueName: \"kubernetes.io/projected/13b254c8-1a1e-40db-b169-9c13a98430f5-kube-api-access-smngq\") pod \"monitoring-plugin-7b7b7bc8bf-ccs8p\" (UID: \"13b254c8-1a1e-40db-b169-9c13a98430f5\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ccs8p" Jul 21 00:21:44.143637 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:44.143575 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:44.179668 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:44.179641 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pz4d4" Jul 21 00:21:44.182469 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:44.182443 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ccs8p" Jul 21 00:21:44.843103 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:44.843071 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ccs8p"] Jul 21 00:21:44.845436 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:21:44.845408 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b254c8_1a1e_40db_b169_9c13a98430f5.slice/crio-aca94026d56e8a10d97b5812ec09f04067064517b59434399b7fa598832bbf3e WatchSource:0}: Error finding container aca94026d56e8a10d97b5812ec09f04067064517b59434399b7fa598832bbf3e: Status 404 returned error can't find the container with id aca94026d56e8a10d97b5812ec09f04067064517b59434399b7fa598832bbf3e Jul 21 00:21:45.225241 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:45.225142 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ccs8p" event={"ID":"13b254c8-1a1e-40db-b169-9c13a98430f5","Type":"ContainerStarted","Data":"aca94026d56e8a10d97b5812ec09f04067064517b59434399b7fa598832bbf3e"} Jul 21 00:21:45.227097 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:45.227071 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" event={"ID":"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e","Type":"ContainerStarted","Data":"dd0ab51961956682c1f2adf3d76dae4485686e992414d65063da49efde99d814"} Jul 21 00:21:45.227207 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:45.227129 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" event={"ID":"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e","Type":"ContainerStarted","Data":"61a600b24a4cd890dc74b6753f4f0964f06861e6f764523598ebe219fb6904ca"} Jul 21 00:21:45.227207 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:45.227155 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" event={"ID":"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e","Type":"ContainerStarted","Data":"091649fdb07b08648f224970a55544ce50fd1d8a9ed5e212c3162580b6f48348"} Jul 21 00:21:45.300697 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:45.300636 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:46.097734 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.097686 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" podUID="a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79" containerName="registry" containerID="cri-o://c559e78ea5bd88c395615c4e1516f2778608b934be8fd9793414d35166503883" gracePeriod=30 Jul 21 00:21:46.233709 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.233667 2580 generic.go:356] "Generic (PLEG): container finished" podID="a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79" containerID="c559e78ea5bd88c395615c4e1516f2778608b934be8fd9793414d35166503883" exitCode=0 Jul 21 00:21:46.234165 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.233828 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" event={"ID":"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79","Type":"ContainerDied","Data":"c559e78ea5bd88c395615c4e1516f2778608b934be8fd9793414d35166503883"} Jul 21 00:21:46.238366 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.238192 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" event={"ID":"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e","Type":"ContainerStarted","Data":"5a65a23da68a080ea9107d7080e5f7dc61f77707e9f7d923a11d62ed7d8edbcd"} Jul 21 00:21:46.238366 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.238251 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" event={"ID":"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e","Type":"ContainerStarted","Data":"1d648bada48eedb1e5a147dfb9556a87131014a50c2ed58ede16a5fd68c881cd"} Jul 21 00:21:46.238366 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.238277 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" event={"ID":"d9783a8f-d1a0-4d04-8452-e8cc6520cf5e","Type":"ContainerStarted","Data":"98e5908a130f3e48fe6e853deeb5292ad8f1e9f7e8eb034fc4960258e5b56715"} Jul 21 00:21:46.238366 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.238344 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:46.261757 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.261710 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" podStartSLOduration=1.172143633 podStartE2EDuration="4.261693606s" totalImagesPullingTime="3.089549973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:21:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:21:46.260069419 +0000 UTC m=+300.475449744" watchObservedRunningTime="2026-07-21 00:21:46.261693606 +0000 UTC m=+300.477073931" Jul 21 00:21:46.553227 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.553197 2580 kubelet.go:1648] "Image garbage collection succeeded" Jul 21 00:21:46.652999 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.652972 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:21:46.728195 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.728164 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-ca-trust-extracted\") pod \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " Jul 21 00:21:46.728379 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.728215 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-bound-sa-token\") pod \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " Jul 21 00:21:46.728379 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.728233 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4f45l\" (UniqueName: \"kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-kube-api-access-4f45l\") pod \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " Jul 21 00:21:46.728379 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.728359 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-tls\") pod \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " Jul 21 00:21:46.728557 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.728399 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-certificates\") pod \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " Jul 21 00:21:46.728557 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.728427 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-trusted-ca\") pod \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " Jul 21 00:21:46.728557 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.728464 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-image-registry-private-configuration\") pod \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " Jul 21 00:21:46.728557 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.728500 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-installation-pull-secrets\") pod \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\" (UID: \"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79\") " Jul 21 00:21:46.728875 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.728844 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79" (UID: "a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:21:46.729024 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.728999 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79" (UID: "a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:21:46.730929 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.730895 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-kube-api-access-4f45l" (OuterVolumeSpecName: "kube-api-access-4f45l") pod "a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79" (UID: "a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79"). InnerVolumeSpecName "kube-api-access-4f45l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:21:46.731251 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.731227 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79" (UID: "a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:21:46.731251 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.731236 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79" (UID: "a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:21:46.731251 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.731246 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79" (UID: "a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:21:46.731450 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.731433 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79" (UID: "a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:21:46.737314 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.737287 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79" (UID: "a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:21:46.804390 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.804357 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:21:46.804517 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.804430 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:21:46.809333 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.809307 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:21:46.829130 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.829095 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f45l\" (UniqueName: \"kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-kube-api-access-4f45l\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:21:46.829130 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.829124 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-tls\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:21:46.829130 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.829134 2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-registry-certificates\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:21:46.829353 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.829144 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-trusted-ca\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:21:46.829353 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.829154 2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-image-registry-private-configuration\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:21:46.829353 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.829163 2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-installation-pull-secrets\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:21:46.829353 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.829172 2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-ca-trust-extracted\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:21:46.829353 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:46.829180 2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79-bound-sa-token\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:21:47.242836 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.242741 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" Jul 21 00:21:47.242836 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.242748 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666c4b4fb-psnt5" event={"ID":"a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79","Type":"ContainerDied","Data":"a1606e1bb5d40934303d129248088e9ebdae66299f730da84aff41915997c880"} Jul 21 00:21:47.243467 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.243445 2580 scope.go:117] "RemoveContainer" containerID="c559e78ea5bd88c395615c4e1516f2778608b934be8fd9793414d35166503883" Jul 21 00:21:47.244355 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.244330 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ccs8p" event={"ID":"13b254c8-1a1e-40db-b169-9c13a98430f5","Type":"ContainerStarted","Data":"ea1d91db943a79867c3ceaf75a7c93e565589d31a2f711e97804d67b8f5d130a"} Jul 21 00:21:47.249050 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.249030 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:21:47.263279 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.263227 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ccs8p" podStartSLOduration=2.533671433 podStartE2EDuration="4.263210559s" totalImagesPullingTime="1.729539126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:21:47.262145513 +0000 UTC m=+301.477525842" watchObservedRunningTime="2026-07-21 00:21:47.263210559 +0000 UTC m=+301.478590883" Jul 21 00:21:47.279587 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.275887 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-666c4b4fb-psnt5"] Jul 21 00:21:47.280504 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.280467 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-666c4b4fb-psnt5"] Jul 21 00:21:47.519852 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.519762 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5ff6c45c9b-c64fn"] Jul 21 00:21:47.520126 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.520107 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79" containerName="registry" Jul 21 00:21:47.520205 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.520129 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79" containerName="registry" Jul 21 00:21:47.520257 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.520205 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79" containerName="registry" Jul 21 00:21:47.524939 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.524916 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:21:47.531613 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.531557 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 21 00:21:47.533481 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.533453 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ff6c45c9b-c64fn"] Jul 21 00:21:47.637985 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.637946 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-console-config\") pod \"console-5ff6c45c9b-c64fn\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:21:47.638168 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.638004 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-trusted-ca-bundle\") pod \"console-5ff6c45c9b-c64fn\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:21:47.638168 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.638087 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gnfq\" (UniqueName: \"kubernetes.io/projected/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-kube-api-access-8gnfq\") pod \"console-5ff6c45c9b-c64fn\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:21:47.638168 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.638125 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-oauth-serving-cert\") pod \"console-5ff6c45c9b-c64fn\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:21:47.638272 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.638170 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-service-ca\") pod \"console-5ff6c45c9b-c64fn\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:21:47.638272 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.638188 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-console-serving-cert\") pod \"console-5ff6c45c9b-c64fn\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:21:47.638272 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.638218 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-console-oauth-config\") pod \"console-5ff6c45c9b-c64fn\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:21:47.739447 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.739399 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gnfq\" (UniqueName: \"kubernetes.io/projected/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-kube-api-access-8gnfq\") pod \"console-5ff6c45c9b-c64fn\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:21:47.739447 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.739452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-oauth-serving-cert\") pod \"console-5ff6c45c9b-c64fn\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:21:47.739568 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.739479 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-service-ca\") pod \"console-5ff6c45c9b-c64fn\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:21:47.739568 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.739497 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-console-serving-cert\") pod \"console-5ff6c45c9b-c64fn\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:21:47.739568 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.739517 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-console-oauth-config\") pod \"console-5ff6c45c9b-c64fn\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:21:47.739568 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.739539 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-console-config\") pod \"console-5ff6c45c9b-c64fn\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:21:47.739768 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.739578 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-trusted-ca-bundle\") pod \"console-5ff6c45c9b-c64fn\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:21:47.740383 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.740351 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-oauth-serving-cert\") pod \"console-5ff6c45c9b-c64fn\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:21:47.740528 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.740351 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-service-ca\") pod \"console-5ff6c45c9b-c64fn\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:21:47.740528 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.740505 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-trusted-ca-bundle\") pod \"console-5ff6c45c9b-c64fn\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:21:47.740650 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.740558 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-console-config\") pod \"console-5ff6c45c9b-c64fn\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:21:47.742143 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.742120 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-console-oauth-config\") pod \"console-5ff6c45c9b-c64fn\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:21:47.742268 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.742248 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-console-serving-cert\") pod \"console-5ff6c45c9b-c64fn\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:21:47.747465 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.747442 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gnfq\" (UniqueName: \"kubernetes.io/projected/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-kube-api-access-8gnfq\") pod \"console-5ff6c45c9b-c64fn\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:21:47.834376 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.834323 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:21:47.885212 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.885173 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:47.945444 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.945410 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:47.969791 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:47.969760 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ff6c45c9b-c64fn"] Jul 21 00:21:47.973029 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:21:47.973002 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af28b65_bbfa_4094_9a1f_e9b08eee08b6.slice/crio-a5b59a0e151c998feedc3967704955e346e14930254ed6e732291dd3db3141b6 WatchSource:0}: Error finding container a5b59a0e151c998feedc3967704955e346e14930254ed6e732291dd3db3141b6: Status 404 returned error can't find the container with id a5b59a0e151c998feedc3967704955e346e14930254ed6e732291dd3db3141b6 Jul 21 00:21:48.249118 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:48.249019 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff6c45c9b-c64fn" event={"ID":"0af28b65-bbfa-4094-9a1f-e9b08eee08b6","Type":"ContainerStarted","Data":"c14de278150de98885dd8f605a0792a2983f62e7a98f273cac8c22354fd4d7ea"} Jul 21 00:21:48.249118 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:48.249108 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff6c45c9b-c64fn" event={"ID":"0af28b65-bbfa-4094-9a1f-e9b08eee08b6","Type":"ContainerStarted","Data":"a5b59a0e151c998feedc3967704955e346e14930254ed6e732291dd3db3141b6"} Jul 21 00:21:48.250107 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:48.250087 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ccs8p" Jul 21 00:21:48.254711 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:48.254687 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-ccs8p" Jul 21 00:21:48.269822 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:48.269769 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5ff6c45c9b-c64fn" podStartSLOduration=1.269752527 podStartE2EDuration="1.269752527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:21:48.268638245 +0000 UTC m=+302.484018570" watchObservedRunningTime="2026-07-21 00:21:48.269752527 +0000 UTC m=+302.485132853" Jul 21 00:21:48.296000 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:48.295969 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79" path="/var/lib/kubelet/pods/a6d0eb64-f2ae-4ecb-9d8d-744c8bd24d79/volumes" Jul 21 00:21:48.701141 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:48.701105 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ff6c45c9b-c64fn"] Jul 21 00:21:50.089403 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:50.089367 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:50.190324 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:50.190290 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:51.923510 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:51.923472 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:51.982512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:51.982475 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:52.068956 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:52.068923 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:52.126337 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:52.126294 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:52.251331 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:52.251261 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-59b7fcc89f-frkvp" Jul 21 00:21:52.291679 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:52.291644 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:21:53.104406 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:53.104371 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:53.204782 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:53.204741 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:53.306673 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:53.306634 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:53.405782 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:53.405702 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:56.067295 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:56.067256 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:56.116838 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:56.116803 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:57.113956 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:57.113921 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:57.167460 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:57.167424 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:21:57.834708 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:57.834672 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:21:59.998173 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:21:59.998138 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85c6d96c66-rq4z4"] Jul 21 00:22:00.110626 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:00.110571 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:22:00.206255 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:00.206221 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:22:00.330088 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:00.330048 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:22:00.429498 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:00.429461 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:22:00.946537 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:00.946498 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:22:01.047860 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:01.047826 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:22:05.186876 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:05.186816 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:22:05.285352 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:05.285321 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:22:11.264102 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:11.264050 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:22:11.315056 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:11.315024 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:22:14.755825 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:14.755785 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:22:14.857950 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:14.857909 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:22:15.276935 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:15.276892 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5ff6c45c9b-c64fn" podUID="0af28b65-bbfa-4094-9a1f-e9b08eee08b6" containerName="console" containerID="cri-o://c14de278150de98885dd8f605a0792a2983f62e7a98f273cac8c22354fd4d7ea" gracePeriod=15 Jul 21 00:22:15.522909 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:15.522884 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:22:15.675678 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:15.675633 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-console-serving-cert\") pod \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " Jul 21 00:22:15.675981 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:15.675699 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gnfq\" (UniqueName: \"kubernetes.io/projected/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-kube-api-access-8gnfq\") pod \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " Jul 21 00:22:15.675981 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:15.675717 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-oauth-serving-cert\") pod \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " Jul 21 00:22:15.675981 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:15.675737 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-console-oauth-config\") pod \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " Jul 21 00:22:15.675981 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:15.675768 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-service-ca\") pod \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " Jul 21 00:22:15.675981 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:15.675797 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-console-config\") pod \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " Jul 21 00:22:15.675981 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:15.675825 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-trusted-ca-bundle\") pod \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\" (UID: \"0af28b65-bbfa-4094-9a1f-e9b08eee08b6\") " Jul 21 00:22:15.676289 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:15.676155 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0af28b65-bbfa-4094-9a1f-e9b08eee08b6" (UID: "0af28b65-bbfa-4094-9a1f-e9b08eee08b6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:22:15.676289 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:15.676192 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-service-ca" (OuterVolumeSpecName: "service-ca") pod "0af28b65-bbfa-4094-9a1f-e9b08eee08b6" (UID: "0af28b65-bbfa-4094-9a1f-e9b08eee08b6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:22:15.676415 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:15.676291 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0af28b65-bbfa-4094-9a1f-e9b08eee08b6" (UID: "0af28b65-bbfa-4094-9a1f-e9b08eee08b6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:22:15.676415 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:15.676412 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-console-config" (OuterVolumeSpecName: "console-config") pod "0af28b65-bbfa-4094-9a1f-e9b08eee08b6" (UID: "0af28b65-bbfa-4094-9a1f-e9b08eee08b6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:22:15.678176 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:15.678134 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0af28b65-bbfa-4094-9a1f-e9b08eee08b6" (UID: "0af28b65-bbfa-4094-9a1f-e9b08eee08b6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:22:15.678176 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:15.678170 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-kube-api-access-8gnfq" (OuterVolumeSpecName: "kube-api-access-8gnfq") pod "0af28b65-bbfa-4094-9a1f-e9b08eee08b6" (UID: "0af28b65-bbfa-4094-9a1f-e9b08eee08b6"). InnerVolumeSpecName "kube-api-access-8gnfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:22:15.678317 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:15.678170 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0af28b65-bbfa-4094-9a1f-e9b08eee08b6" (UID: "0af28b65-bbfa-4094-9a1f-e9b08eee08b6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:22:15.776434 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:15.776394 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-service-ca\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:22:15.776434 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:15.776428 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-console-config\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:22:15.776434 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:15.776437 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-trusted-ca-bundle\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:22:15.776919 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:15.776447 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-console-serving-cert\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:22:15.776919 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:15.776457 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gnfq\" (UniqueName: \"kubernetes.io/projected/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-kube-api-access-8gnfq\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:22:15.776919 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:15.776465 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-oauth-serving-cert\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:22:15.776919 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:15.776482 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0af28b65-bbfa-4094-9a1f-e9b08eee08b6-console-oauth-config\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:22:16.348864 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:16.348831 2580 generic.go:356] "Generic (PLEG): container finished" podID="0af28b65-bbfa-4094-9a1f-e9b08eee08b6" containerID="c14de278150de98885dd8f605a0792a2983f62e7a98f273cac8c22354fd4d7ea" exitCode=0 Jul 21 00:22:16.348864 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:16.348877 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff6c45c9b-c64fn" event={"ID":"0af28b65-bbfa-4094-9a1f-e9b08eee08b6","Type":"ContainerDied","Data":"c14de278150de98885dd8f605a0792a2983f62e7a98f273cac8c22354fd4d7ea"} Jul 21 00:22:16.349130 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:16.348885 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff6c45c9b-c64fn" Jul 21 00:22:16.349130 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:16.348917 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff6c45c9b-c64fn" event={"ID":"0af28b65-bbfa-4094-9a1f-e9b08eee08b6","Type":"ContainerDied","Data":"a5b59a0e151c998feedc3967704955e346e14930254ed6e732291dd3db3141b6"} Jul 21 00:22:16.349130 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:16.348937 2580 scope.go:117] "RemoveContainer" containerID="c14de278150de98885dd8f605a0792a2983f62e7a98f273cac8c22354fd4d7ea" Jul 21 00:22:16.357094 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:16.357072 2580 scope.go:117] "RemoveContainer" containerID="c14de278150de98885dd8f605a0792a2983f62e7a98f273cac8c22354fd4d7ea" Jul 21 00:22:16.357410 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:22:16.357387 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c14de278150de98885dd8f605a0792a2983f62e7a98f273cac8c22354fd4d7ea\": container with ID starting with c14de278150de98885dd8f605a0792a2983f62e7a98f273cac8c22354fd4d7ea not found: ID does not exist" containerID="c14de278150de98885dd8f605a0792a2983f62e7a98f273cac8c22354fd4d7ea" Jul 21 00:22:16.357494 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:16.357420 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c14de278150de98885dd8f605a0792a2983f62e7a98f273cac8c22354fd4d7ea"} err="failed to get container status \"c14de278150de98885dd8f605a0792a2983f62e7a98f273cac8c22354fd4d7ea\": rpc error: code = NotFound desc = could not find container \"c14de278150de98885dd8f605a0792a2983f62e7a98f273cac8c22354fd4d7ea\": container with ID starting with c14de278150de98885dd8f605a0792a2983f62e7a98f273cac8c22354fd4d7ea not found: ID does not exist" Jul 21 00:22:16.366257 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:16.366226 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ff6c45c9b-c64fn"] Jul 21 00:22:16.369588 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:16.369564 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5ff6c45c9b-c64fn"] Jul 21 00:22:18.295110 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:18.295076 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0af28b65-bbfa-4094-9a1f-e9b08eee08b6" path="/var/lib/kubelet/pods/0af28b65-bbfa-4094-9a1f-e9b08eee08b6/volumes" Jul 21 00:22:25.019488 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.019441 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-85c6d96c66-rq4z4" podUID="f1e5fc8a-8a6e-4627-adcf-953ad08a11d1" containerName="console" containerID="cri-o://21a01b092f4d659cf233a17e53ed6c31553028a48ddc6004920d8936c2c4f3c6" gracePeriod=15 Jul 21 00:22:25.261063 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.261037 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:22:25.358592 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.358551 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-console-config\") pod \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\" (UID: \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\") " Jul 21 00:22:25.358786 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.358636 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-oauth-serving-cert\") pod \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\" (UID: \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\") " Jul 21 00:22:25.358786 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.358729 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-service-ca\") pod \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\" (UID: \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\") " Jul 21 00:22:25.358786 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.358777 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-console-oauth-config\") pod \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\" (UID: \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\") " Jul 21 00:22:25.358926 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.358803 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-console-serving-cert\") pod \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\" (UID: \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\") " Jul 21 00:22:25.358926 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.358853 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xrrn\" (UniqueName: \"kubernetes.io/projected/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-kube-api-access-8xrrn\") pod \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\" (UID: \"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1\") " Jul 21 00:22:25.359049 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.359024 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f1e5fc8a-8a6e-4627-adcf-953ad08a11d1" (UID: "f1e5fc8a-8a6e-4627-adcf-953ad08a11d1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:22:25.359049 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.359032 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-console-config" (OuterVolumeSpecName: "console-config") pod "f1e5fc8a-8a6e-4627-adcf-953ad08a11d1" (UID: "f1e5fc8a-8a6e-4627-adcf-953ad08a11d1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:22:25.359153 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.359084 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-service-ca" (OuterVolumeSpecName: "service-ca") pod "f1e5fc8a-8a6e-4627-adcf-953ad08a11d1" (UID: "f1e5fc8a-8a6e-4627-adcf-953ad08a11d1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:22:25.361143 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.361114 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-kube-api-access-8xrrn" (OuterVolumeSpecName: "kube-api-access-8xrrn") pod "f1e5fc8a-8a6e-4627-adcf-953ad08a11d1" (UID: "f1e5fc8a-8a6e-4627-adcf-953ad08a11d1"). InnerVolumeSpecName "kube-api-access-8xrrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:22:25.361267 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.361167 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f1e5fc8a-8a6e-4627-adcf-953ad08a11d1" (UID: "f1e5fc8a-8a6e-4627-adcf-953ad08a11d1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:22:25.361267 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.361251 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f1e5fc8a-8a6e-4627-adcf-953ad08a11d1" (UID: "f1e5fc8a-8a6e-4627-adcf-953ad08a11d1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:22:25.381538 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.381507 2580 generic.go:356] "Generic (PLEG): container finished" podID="f1e5fc8a-8a6e-4627-adcf-953ad08a11d1" containerID="21a01b092f4d659cf233a17e53ed6c31553028a48ddc6004920d8936c2c4f3c6" exitCode=0 Jul 21 00:22:25.381714 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.381553 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c6d96c66-rq4z4" Jul 21 00:22:25.381714 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.381612 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c6d96c66-rq4z4" event={"ID":"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1","Type":"ContainerDied","Data":"21a01b092f4d659cf233a17e53ed6c31553028a48ddc6004920d8936c2c4f3c6"} Jul 21 00:22:25.381714 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.381670 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c6d96c66-rq4z4" event={"ID":"f1e5fc8a-8a6e-4627-adcf-953ad08a11d1","Type":"ContainerDied","Data":"04dfa5de582635a670513f07ffa7940c437375ef31944d8514476d4d3c340bd6"} Jul 21 00:22:25.381714 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.381697 2580 scope.go:117] "RemoveContainer" containerID="21a01b092f4d659cf233a17e53ed6c31553028a48ddc6004920d8936c2c4f3c6" Jul 21 00:22:25.391924 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.391765 2580 scope.go:117] "RemoveContainer" containerID="21a01b092f4d659cf233a17e53ed6c31553028a48ddc6004920d8936c2c4f3c6" Jul 21 00:22:25.392285 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:22:25.392250 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a01b092f4d659cf233a17e53ed6c31553028a48ddc6004920d8936c2c4f3c6\": container with ID starting with 21a01b092f4d659cf233a17e53ed6c31553028a48ddc6004920d8936c2c4f3c6 not found: ID does not exist" containerID="21a01b092f4d659cf233a17e53ed6c31553028a48ddc6004920d8936c2c4f3c6" Jul 21 00:22:25.392385 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.392290 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a01b092f4d659cf233a17e53ed6c31553028a48ddc6004920d8936c2c4f3c6"} err="failed to get container status \"21a01b092f4d659cf233a17e53ed6c31553028a48ddc6004920d8936c2c4f3c6\": rpc error: code = NotFound desc = could not find container \"21a01b092f4d659cf233a17e53ed6c31553028a48ddc6004920d8936c2c4f3c6\": container with ID starting with 21a01b092f4d659cf233a17e53ed6c31553028a48ddc6004920d8936c2c4f3c6 not found: ID does not exist" Jul 21 00:22:25.404935 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.404905 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85c6d96c66-rq4z4"] Jul 21 00:22:25.407978 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.407950 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85c6d96c66-rq4z4"] Jul 21 00:22:25.459569 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.459527 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xrrn\" (UniqueName: \"kubernetes.io/projected/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-kube-api-access-8xrrn\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:22:25.459569 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.459557 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-console-config\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:22:25.459569 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.459566 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-oauth-serving-cert\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:22:25.459569 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.459575 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-service-ca\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:22:25.459569 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.459583 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-console-oauth-config\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:22:25.459881 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:25.459592 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1-console-serving-cert\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:22:26.295639 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:26.295584 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1e5fc8a-8a6e-4627-adcf-953ad08a11d1" path="/var/lib/kubelet/pods/f1e5fc8a-8a6e-4627-adcf-953ad08a11d1/volumes" Jul 21 00:22:29.671667 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:29.671636 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:22:29.725844 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:29.725814 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:22:30.958261 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:30.958219 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-br7vh_5046e73d-17d6-4282-9387-c665c60f7ca9/serve-healthcheck-canary/0.log" Jul 21 00:22:34.293723 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:34.293688 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:22:34.393678 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:34.393631 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:22:34.447968 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:34.447936 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:22:55.174465 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:55.174423 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:22:55.277254 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:55.277217 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:22:58.349576 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:58.349541 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:22:58.433494 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:22:58.433454 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:23:00.113655 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:00.113578 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs\") pod \"network-metrics-daemon-n8sh6\" (UID: \"982c215c-29b3-45ac-8525-03c61f27e931\") " pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:23:00.116052 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:00.116027 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/982c215c-29b3-45ac-8525-03c61f27e931-metrics-certs\") pod \"network-metrics-daemon-n8sh6\" (UID: \"982c215c-29b3-45ac-8525-03c61f27e931\") " pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:23:00.394556 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:00.394475 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ztz7f\"" Jul 21 00:23:00.402669 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:00.402645 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n8sh6" Jul 21 00:23:00.548628 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:00.548199 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n8sh6"] Jul 21 00:23:00.557139 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:00.557114 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 00:23:01.510207 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:01.510167 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n8sh6" event={"ID":"982c215c-29b3-45ac-8525-03c61f27e931","Type":"ContainerStarted","Data":"448c03141fb48c67e9a7ab02374c53e1004feae471f9ea8d16718fe0b795d82c"} Jul 21 00:23:02.177573 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.177539 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68df77886f-ghr8l"] Jul 21 00:23:02.177848 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.177827 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1e5fc8a-8a6e-4627-adcf-953ad08a11d1" containerName="console" Jul 21 00:23:02.177848 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.177843 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1e5fc8a-8a6e-4627-adcf-953ad08a11d1" containerName="console" Jul 21 00:23:02.178039 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.177859 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0af28b65-bbfa-4094-9a1f-e9b08eee08b6" containerName="console" Jul 21 00:23:02.178039 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.177866 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0af28b65-bbfa-4094-9a1f-e9b08eee08b6" containerName="console" Jul 21 00:23:02.178039 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.177920 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0af28b65-bbfa-4094-9a1f-e9b08eee08b6" containerName="console" Jul 21 00:23:02.178039 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.177930 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1e5fc8a-8a6e-4627-adcf-953ad08a11d1" containerName="console" Jul 21 00:23:02.179919 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.179901 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:02.182544 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.182526 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 21 00:23:02.183655 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.183630 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 21 00:23:02.183655 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.183644 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 21 00:23:02.183847 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.183646 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 21 00:23:02.183847 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.183675 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 21 00:23:02.184000 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.183984 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 21 00:23:02.184054 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.184040 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ks6p9\"" Jul 21 00:23:02.184091 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.184080 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 21 00:23:02.190179 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.190151 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 21 00:23:02.197383 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.197350 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68df77886f-ghr8l"] Jul 21 00:23:02.229573 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.229532 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8f2e8f7-b292-4100-af63-85dd51625ac5-console-oauth-config\") pod \"console-68df77886f-ghr8l\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:02.229573 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.229568 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8f2e8f7-b292-4100-af63-85dd51625ac5-trusted-ca-bundle\") pod \"console-68df77886f-ghr8l\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:02.229856 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.229587 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8f2e8f7-b292-4100-af63-85dd51625ac5-oauth-serving-cert\") pod \"console-68df77886f-ghr8l\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:02.229856 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.229631 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm878\" (UniqueName: \"kubernetes.io/projected/b8f2e8f7-b292-4100-af63-85dd51625ac5-kube-api-access-cm878\") pod \"console-68df77886f-ghr8l\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:02.229856 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.229713 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8f2e8f7-b292-4100-af63-85dd51625ac5-console-serving-cert\") pod \"console-68df77886f-ghr8l\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:02.229856 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.229797 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8f2e8f7-b292-4100-af63-85dd51625ac5-console-config\") pod \"console-68df77886f-ghr8l\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:02.229856 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.229847 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8f2e8f7-b292-4100-af63-85dd51625ac5-service-ca\") pod \"console-68df77886f-ghr8l\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:02.261033 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.261001 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:23:02.330441 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.330403 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8f2e8f7-b292-4100-af63-85dd51625ac5-console-config\") pod \"console-68df77886f-ghr8l\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:02.330441 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.330449 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8f2e8f7-b292-4100-af63-85dd51625ac5-service-ca\") pod \"console-68df77886f-ghr8l\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:02.330747 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.330470 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8f2e8f7-b292-4100-af63-85dd51625ac5-console-oauth-config\") pod \"console-68df77886f-ghr8l\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:02.330747 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.330497 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8f2e8f7-b292-4100-af63-85dd51625ac5-trusted-ca-bundle\") pod \"console-68df77886f-ghr8l\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:02.330747 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.330522 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8f2e8f7-b292-4100-af63-85dd51625ac5-oauth-serving-cert\") pod \"console-68df77886f-ghr8l\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:02.330747 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.330549 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm878\" (UniqueName: \"kubernetes.io/projected/b8f2e8f7-b292-4100-af63-85dd51625ac5-kube-api-access-cm878\") pod \"console-68df77886f-ghr8l\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:02.330747 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.330580 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8f2e8f7-b292-4100-af63-85dd51625ac5-console-serving-cert\") pod \"console-68df77886f-ghr8l\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:02.331250 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.331222 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8f2e8f7-b292-4100-af63-85dd51625ac5-console-config\") pod \"console-68df77886f-ghr8l\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:02.331370 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.331231 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8f2e8f7-b292-4100-af63-85dd51625ac5-service-ca\") pod \"console-68df77886f-ghr8l\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:02.331370 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.331350 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8f2e8f7-b292-4100-af63-85dd51625ac5-oauth-serving-cert\") pod \"console-68df77886f-ghr8l\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:02.331445 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.331416 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8f2e8f7-b292-4100-af63-85dd51625ac5-trusted-ca-bundle\") pod \"console-68df77886f-ghr8l\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:02.333676 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.333652 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8f2e8f7-b292-4100-af63-85dd51625ac5-console-serving-cert\") pod \"console-68df77886f-ghr8l\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:02.333777 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.333730 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8f2e8f7-b292-4100-af63-85dd51625ac5-console-oauth-config\") pod \"console-68df77886f-ghr8l\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:02.340076 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.340056 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm878\" (UniqueName: \"kubernetes.io/projected/b8f2e8f7-b292-4100-af63-85dd51625ac5-kube-api-access-cm878\") pod \"console-68df77886f-ghr8l\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:02.411131 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.411090 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:23:02.489797 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.489709 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:02.515713 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.515673 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n8sh6" event={"ID":"982c215c-29b3-45ac-8525-03c61f27e931","Type":"ContainerStarted","Data":"41dabd20a39624c673dc74cc247477f6654379fc8fe87543888500d23ab179da"} Jul 21 00:23:02.516068 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.515748 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n8sh6" event={"ID":"982c215c-29b3-45ac-8525-03c61f27e931","Type":"ContainerStarted","Data":"10090055173beb416ac23d984cfb37b6be692e58ba0cc139fa121abb0c21a163"} Jul 21 00:23:02.534409 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.534343 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-n8sh6" podStartSLOduration=375.388338704 podStartE2EDuration="6m16.534319731s" totalImagesPullingTime="1.145981027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:23:02.533087238 +0000 UTC m=+376.748467587" watchObservedRunningTime="2026-07-21 00:23:02.534319731 +0000 UTC m=+376.749700060" Jul 21 00:23:02.663887 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.663859 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68df77886f-ghr8l"] Jul 21 00:23:02.667131 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:23:02.667094 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f2e8f7_b292_4100_af63_85dd51625ac5.slice/crio-a4aab7967035c014fe6573b5af41a0def38befd63af8f43faaeb9f331561096c WatchSource:0}: Error finding container a4aab7967035c014fe6573b5af41a0def38befd63af8f43faaeb9f331561096c: Status 404 returned error can't find the container with id a4aab7967035c014fe6573b5af41a0def38befd63af8f43faaeb9f331561096c Jul 21 00:23:02.673481 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.673458 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:23:02.784660 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:02.784397 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:23:03.521211 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:03.521171 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68df77886f-ghr8l" event={"ID":"b8f2e8f7-b292-4100-af63-85dd51625ac5","Type":"ContainerStarted","Data":"b115e29434d08e0b69d303f4162be4fd5816d724aab3f28b04b74545902c616e"} Jul 21 00:23:03.521719 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:03.521223 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68df77886f-ghr8l" event={"ID":"b8f2e8f7-b292-4100-af63-85dd51625ac5","Type":"ContainerStarted","Data":"a4aab7967035c014fe6573b5af41a0def38befd63af8f43faaeb9f331561096c"} Jul 21 00:23:03.556385 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:03.556328 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68df77886f-ghr8l" podStartSLOduration=1.556311193 podStartE2EDuration="1.556311193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:23:03.556007183 +0000 UTC m=+377.771387509" watchObservedRunningTime="2026-07-21 00:23:03.556311193 +0000 UTC m=+377.771691518" Jul 21 00:23:07.046861 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:07.046818 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:23:07.117284 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:07.117247 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:23:08.584284 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:08.584243 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:23:08.640847 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:08.640810 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:23:12.490813 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:12.490776 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:12.491214 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:12.490833 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:12.495645 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:12.495619 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:12.564112 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:12.564084 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:23:12.779786 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:12.779676 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:23:12.879084 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:12.879044 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:23:12.967548 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:12.967505 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:23:14.681128 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:14.681085 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:23:14.777041 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:23:14.776998 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:24:01.271252 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:01.271211 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:24:01.358119 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:01.358082 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:24:11.357094 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:11.356989 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:24:11.487055 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:11.487016 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:24:12.024887 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:12.024848 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8"] Jul 21 00:24:12.027347 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:12.027323 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8" Jul 21 00:24:12.031401 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:12.031377 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 00:24:12.032483 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:12.032464 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 00:24:12.032661 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:12.032645 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4sl9\"" Jul 21 00:24:12.037307 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:12.037282 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8"] Jul 21 00:24:12.130488 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:12.130441 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38b1a24c-e8fe-410a-ae1f-d4a048270206-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8\" (UID: \"38b1a24c-e8fe-410a-ae1f-d4a048270206\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8" Jul 21 00:24:12.130488 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:12.130487 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rch29\" (UniqueName: \"kubernetes.io/projected/38b1a24c-e8fe-410a-ae1f-d4a048270206-kube-api-access-rch29\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8\" (UID: \"38b1a24c-e8fe-410a-ae1f-d4a048270206\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8" Jul 21 00:24:12.130733 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:12.130513 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38b1a24c-e8fe-410a-ae1f-d4a048270206-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8\" (UID: \"38b1a24c-e8fe-410a-ae1f-d4a048270206\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8" Jul 21 00:24:12.231814 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:12.231778 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38b1a24c-e8fe-410a-ae1f-d4a048270206-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8\" (UID: \"38b1a24c-e8fe-410a-ae1f-d4a048270206\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8" Jul 21 00:24:12.231814 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:12.231814 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rch29\" (UniqueName: \"kubernetes.io/projected/38b1a24c-e8fe-410a-ae1f-d4a048270206-kube-api-access-rch29\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8\" (UID: \"38b1a24c-e8fe-410a-ae1f-d4a048270206\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8" Jul 21 00:24:12.232065 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:12.231835 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38b1a24c-e8fe-410a-ae1f-d4a048270206-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8\" (UID: \"38b1a24c-e8fe-410a-ae1f-d4a048270206\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8" Jul 21 00:24:12.232181 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:12.232163 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38b1a24c-e8fe-410a-ae1f-d4a048270206-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8\" (UID: \"38b1a24c-e8fe-410a-ae1f-d4a048270206\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8" Jul 21 00:24:12.232263 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:12.232240 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38b1a24c-e8fe-410a-ae1f-d4a048270206-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8\" (UID: \"38b1a24c-e8fe-410a-ae1f-d4a048270206\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8" Jul 21 00:24:12.241453 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:12.241430 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rch29\" (UniqueName: \"kubernetes.io/projected/38b1a24c-e8fe-410a-ae1f-d4a048270206-kube-api-access-rch29\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8\" (UID: \"38b1a24c-e8fe-410a-ae1f-d4a048270206\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8" Jul 21 00:24:12.337047 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:12.337008 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8" Jul 21 00:24:12.471407 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:12.471371 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8"] Jul 21 00:24:12.475070 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:24:12.475039 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b1a24c_e8fe_410a_ae1f_d4a048270206.slice/crio-8e2911152b635aaf1bf1ff73fef8d13086e4eca0eca709b9580d18fca3138a35 WatchSource:0}: Error finding container 8e2911152b635aaf1bf1ff73fef8d13086e4eca0eca709b9580d18fca3138a35: Status 404 returned error can't find the container with id 8e2911152b635aaf1bf1ff73fef8d13086e4eca0eca709b9580d18fca3138a35 Jul 21 00:24:12.761504 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:12.761413 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8" event={"ID":"38b1a24c-e8fe-410a-ae1f-d4a048270206","Type":"ContainerStarted","Data":"8e2911152b635aaf1bf1ff73fef8d13086e4eca0eca709b9580d18fca3138a35"} Jul 21 00:24:19.331938 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:19.331891 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:24:19.793899 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:19.793816 2580 generic.go:356] "Generic (PLEG): container finished" podID="38b1a24c-e8fe-410a-ae1f-d4a048270206" containerID="cbaaa958ae3daeb8ca869d6ebe094e90bf9def42fc280d0bc7b4dcf93c5ed09f" exitCode=0 Jul 21 00:24:19.794044 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:19.793900 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8" event={"ID":"38b1a24c-e8fe-410a-ae1f-d4a048270206","Type":"ContainerDied","Data":"cbaaa958ae3daeb8ca869d6ebe094e90bf9def42fc280d0bc7b4dcf93c5ed09f"} Jul 21 00:24:21.803470 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:21.803371 2580 generic.go:356] "Generic (PLEG): container finished" podID="38b1a24c-e8fe-410a-ae1f-d4a048270206" containerID="8944447d7ca35781cd5a54f9eecf15ea20554c6a6309558a4414e66053ee1b98" exitCode=0 Jul 21 00:24:21.803470 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:21.803449 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8" event={"ID":"38b1a24c-e8fe-410a-ae1f-d4a048270206","Type":"ContainerDied","Data":"8944447d7ca35781cd5a54f9eecf15ea20554c6a6309558a4414e66053ee1b98"} Jul 21 00:24:29.700138 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:29.700081 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:24:29.839447 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:29.839412 2580 generic.go:356] "Generic (PLEG): container finished" podID="38b1a24c-e8fe-410a-ae1f-d4a048270206" containerID="fb4d1f96d0fd9475196cc339767de0ba66bb1030670acf1b1283dc24c4bee8a5" exitCode=0 Jul 21 00:24:29.839636 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:29.839499 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8" event={"ID":"38b1a24c-e8fe-410a-ae1f-d4a048270206","Type":"ContainerDied","Data":"fb4d1f96d0fd9475196cc339767de0ba66bb1030670acf1b1283dc24c4bee8a5"} Jul 21 00:24:30.965005 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:30.964979 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8" Jul 21 00:24:31.093930 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:31.093834 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rch29\" (UniqueName: \"kubernetes.io/projected/38b1a24c-e8fe-410a-ae1f-d4a048270206-kube-api-access-rch29\") pod \"38b1a24c-e8fe-410a-ae1f-d4a048270206\" (UID: \"38b1a24c-e8fe-410a-ae1f-d4a048270206\") " Jul 21 00:24:31.093930 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:31.093890 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38b1a24c-e8fe-410a-ae1f-d4a048270206-bundle\") pod \"38b1a24c-e8fe-410a-ae1f-d4a048270206\" (UID: \"38b1a24c-e8fe-410a-ae1f-d4a048270206\") " Jul 21 00:24:31.093930 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:31.093925 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38b1a24c-e8fe-410a-ae1f-d4a048270206-util\") pod \"38b1a24c-e8fe-410a-ae1f-d4a048270206\" (UID: \"38b1a24c-e8fe-410a-ae1f-d4a048270206\") " Jul 21 00:24:31.094647 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:31.094590 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38b1a24c-e8fe-410a-ae1f-d4a048270206-bundle" (OuterVolumeSpecName: "bundle") pod "38b1a24c-e8fe-410a-ae1f-d4a048270206" (UID: "38b1a24c-e8fe-410a-ae1f-d4a048270206"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:24:31.096264 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:31.096241 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38b1a24c-e8fe-410a-ae1f-d4a048270206-kube-api-access-rch29" (OuterVolumeSpecName: "kube-api-access-rch29") pod "38b1a24c-e8fe-410a-ae1f-d4a048270206" (UID: "38b1a24c-e8fe-410a-ae1f-d4a048270206"). InnerVolumeSpecName "kube-api-access-rch29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:24:31.098573 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:31.098550 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38b1a24c-e8fe-410a-ae1f-d4a048270206-util" (OuterVolumeSpecName: "util") pod "38b1a24c-e8fe-410a-ae1f-d4a048270206" (UID: "38b1a24c-e8fe-410a-ae1f-d4a048270206"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:24:31.195141 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:31.195099 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38b1a24c-e8fe-410a-ae1f-d4a048270206-bundle\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:24:31.195141 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:31.195127 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38b1a24c-e8fe-410a-ae1f-d4a048270206-util\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:24:31.195141 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:31.195136 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rch29\" (UniqueName: \"kubernetes.io/projected/38b1a24c-e8fe-410a-ae1f-d4a048270206-kube-api-access-rch29\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:24:31.848731 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:31.848698 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8" Jul 21 00:24:31.848731 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:31.848698 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675407gxb8" event={"ID":"38b1a24c-e8fe-410a-ae1f-d4a048270206","Type":"ContainerDied","Data":"8e2911152b635aaf1bf1ff73fef8d13086e4eca0eca709b9580d18fca3138a35"} Jul 21 00:24:31.849015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:31.848784 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e2911152b635aaf1bf1ff73fef8d13086e4eca0eca709b9580d18fca3138a35" Jul 21 00:24:33.122563 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:33.122528 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:24:33.973547 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:33.973506 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:24:35.065797 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:35.065762 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ggdtn"] Jul 21 00:24:35.066178 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:35.066006 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38b1a24c-e8fe-410a-ae1f-d4a048270206" containerName="pull" Jul 21 00:24:35.066178 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:35.066017 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="38b1a24c-e8fe-410a-ae1f-d4a048270206" containerName="pull" Jul 21 00:24:35.066178 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:35.066034 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38b1a24c-e8fe-410a-ae1f-d4a048270206" containerName="extract" Jul 21 00:24:35.066178 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:35.066040 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="38b1a24c-e8fe-410a-ae1f-d4a048270206" containerName="extract" Jul 21 00:24:35.066178 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:35.066046 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38b1a24c-e8fe-410a-ae1f-d4a048270206" containerName="util" Jul 21 00:24:35.066178 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:35.066051 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="38b1a24c-e8fe-410a-ae1f-d4a048270206" containerName="util" Jul 21 00:24:35.066178 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:35.066093 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="38b1a24c-e8fe-410a-ae1f-d4a048270206" containerName="extract" Jul 21 00:24:35.113385 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:35.113346 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ggdtn"] Jul 21 00:24:35.113552 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:35.113467 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ggdtn" Jul 21 00:24:35.117811 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:35.117787 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-xmq4c\"" Jul 21 00:24:35.117970 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:35.117830 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 21 00:24:35.118295 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:35.118280 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 21 00:24:35.228120 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:35.228083 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24c2f5c1-d2ce-45da-899e-e5fcb3d2d5a0-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-ggdtn\" (UID: \"24c2f5c1-d2ce-45da-899e-e5fcb3d2d5a0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ggdtn" Jul 21 00:24:35.228120 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:35.228131 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5pw2\" (UniqueName: \"kubernetes.io/projected/24c2f5c1-d2ce-45da-899e-e5fcb3d2d5a0-kube-api-access-r5pw2\") pod \"cert-manager-operator-controller-manager-55c444bf86-ggdtn\" (UID: \"24c2f5c1-d2ce-45da-899e-e5fcb3d2d5a0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ggdtn" Jul 21 00:24:35.328558 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:35.328463 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24c2f5c1-d2ce-45da-899e-e5fcb3d2d5a0-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-ggdtn\" (UID: \"24c2f5c1-d2ce-45da-899e-e5fcb3d2d5a0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ggdtn" Jul 21 00:24:35.328558 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:35.328509 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5pw2\" (UniqueName: \"kubernetes.io/projected/24c2f5c1-d2ce-45da-899e-e5fcb3d2d5a0-kube-api-access-r5pw2\") pod \"cert-manager-operator-controller-manager-55c444bf86-ggdtn\" (UID: \"24c2f5c1-d2ce-45da-899e-e5fcb3d2d5a0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ggdtn" Jul 21 00:24:35.328947 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:35.328920 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24c2f5c1-d2ce-45da-899e-e5fcb3d2d5a0-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-ggdtn\" (UID: \"24c2f5c1-d2ce-45da-899e-e5fcb3d2d5a0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ggdtn" Jul 21 00:24:35.337764 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:35.337734 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5pw2\" (UniqueName: \"kubernetes.io/projected/24c2f5c1-d2ce-45da-899e-e5fcb3d2d5a0-kube-api-access-r5pw2\") pod \"cert-manager-operator-controller-manager-55c444bf86-ggdtn\" (UID: \"24c2f5c1-d2ce-45da-899e-e5fcb3d2d5a0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ggdtn" Jul 21 00:24:35.422239 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:35.422193 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ggdtn" Jul 21 00:24:35.583209 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:35.583183 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ggdtn"] Jul 21 00:24:35.586309 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:24:35.586279 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c2f5c1_d2ce_45da_899e_e5fcb3d2d5a0.slice/crio-1b399b415b742b5fbe5107e36d4637d21aa798835b309cb94fd36182c38b125c WatchSource:0}: Error finding container 1b399b415b742b5fbe5107e36d4637d21aa798835b309cb94fd36182c38b125c: Status 404 returned error can't find the container with id 1b399b415b742b5fbe5107e36d4637d21aa798835b309cb94fd36182c38b125c Jul 21 00:24:35.867761 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:35.867665 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ggdtn" event={"ID":"24c2f5c1-d2ce-45da-899e-e5fcb3d2d5a0","Type":"ContainerStarted","Data":"1b399b415b742b5fbe5107e36d4637d21aa798835b309cb94fd36182c38b125c"} Jul 21 00:24:36.649860 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:36.649803 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:24:36.749659 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:36.749611 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:24:38.757472 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:38.757436 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:24:38.874159 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:38.874122 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:24:38.885177 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:38.885130 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ggdtn" event={"ID":"24c2f5c1-d2ce-45da-899e-e5fcb3d2d5a0","Type":"ContainerStarted","Data":"2be123ea99b8ac17d4356a39992c299e0d4a03fdc36f6f1f15f898c992423cdf"} Jul 21 00:24:38.906998 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:38.906943 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ggdtn" podStartSLOduration=1.6732247550000001 podStartE2EDuration="3.906928198s" totalImagesPullingTime="2.233703443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:24:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:24:38.905543556 +0000 UTC m=+473.120923883" watchObservedRunningTime="2026-07-21 00:24:38.906928198 +0000 UTC m=+473.122308523" Jul 21 00:24:39.301770 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:39.301726 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:24:39.443716 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:39.443682 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:24:39.484364 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:39.484326 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974"] Jul 21 00:24:39.509706 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:39.509676 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974"] Jul 21 00:24:39.509890 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:39.509801 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974" Jul 21 00:24:39.514225 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:39.514197 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 00:24:39.514495 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:39.514480 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 00:24:39.514647 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:39.514633 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4sl9\"" Jul 21 00:24:39.661467 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:39.661427 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8be91b06-5f84-4851-ba53-f7ef63f89b53-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974\" (UID: \"8be91b06-5f84-4851-ba53-f7ef63f89b53\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974" Jul 21 00:24:39.661684 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:39.661493 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5rms\" (UniqueName: \"kubernetes.io/projected/8be91b06-5f84-4851-ba53-f7ef63f89b53-kube-api-access-w5rms\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974\" (UID: \"8be91b06-5f84-4851-ba53-f7ef63f89b53\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974" Jul 21 00:24:39.661684 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:39.661540 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8be91b06-5f84-4851-ba53-f7ef63f89b53-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974\" (UID: \"8be91b06-5f84-4851-ba53-f7ef63f89b53\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974" Jul 21 00:24:39.762842 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:39.762791 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8be91b06-5f84-4851-ba53-f7ef63f89b53-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974\" (UID: \"8be91b06-5f84-4851-ba53-f7ef63f89b53\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974" Jul 21 00:24:39.763262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:39.762901 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5rms\" (UniqueName: \"kubernetes.io/projected/8be91b06-5f84-4851-ba53-f7ef63f89b53-kube-api-access-w5rms\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974\" (UID: \"8be91b06-5f84-4851-ba53-f7ef63f89b53\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974" Jul 21 00:24:39.763262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:39.762947 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8be91b06-5f84-4851-ba53-f7ef63f89b53-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974\" (UID: \"8be91b06-5f84-4851-ba53-f7ef63f89b53\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974" Jul 21 00:24:39.763343 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:39.763263 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8be91b06-5f84-4851-ba53-f7ef63f89b53-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974\" (UID: \"8be91b06-5f84-4851-ba53-f7ef63f89b53\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974" Jul 21 00:24:39.763343 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:39.763272 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8be91b06-5f84-4851-ba53-f7ef63f89b53-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974\" (UID: \"8be91b06-5f84-4851-ba53-f7ef63f89b53\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974" Jul 21 00:24:39.771421 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:39.771389 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5rms\" (UniqueName: \"kubernetes.io/projected/8be91b06-5f84-4851-ba53-f7ef63f89b53-kube-api-access-w5rms\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974\" (UID: \"8be91b06-5f84-4851-ba53-f7ef63f89b53\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974" Jul 21 00:24:39.819286 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:39.819243 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974" Jul 21 00:24:39.956294 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:39.956199 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974"] Jul 21 00:24:39.958991 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:24:39.958967 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be91b06_5f84_4851_ba53_f7ef63f89b53.slice/crio-01bd51324e595a4c456f8e3e62bcb3a791fa847649aa089833ef24239a62c6d2 WatchSource:0}: Error finding container 01bd51324e595a4c456f8e3e62bcb3a791fa847649aa089833ef24239a62c6d2: Status 404 returned error can't find the container with id 01bd51324e595a4c456f8e3e62bcb3a791fa847649aa089833ef24239a62c6d2 Jul 21 00:24:40.776202 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:40.776151 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:24:40.895179 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:40.895146 2580 generic.go:356] "Generic (PLEG): container finished" podID="8be91b06-5f84-4851-ba53-f7ef63f89b53" containerID="0e6a76015d650c58e1a09cd0817e623ddb54a76f183de0b253a81b481d02fdd6" exitCode=0 Jul 21 00:24:40.895354 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:40.895216 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974" event={"ID":"8be91b06-5f84-4851-ba53-f7ef63f89b53","Type":"ContainerDied","Data":"0e6a76015d650c58e1a09cd0817e623ddb54a76f183de0b253a81b481d02fdd6"} Jul 21 00:24:40.895354 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:40.895259 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974" event={"ID":"8be91b06-5f84-4851-ba53-f7ef63f89b53","Type":"ContainerStarted","Data":"01bd51324e595a4c456f8e3e62bcb3a791fa847649aa089833ef24239a62c6d2"} Jul 21 00:24:41.178364 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:41.178331 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:24:42.906837 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:42.906806 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974" event={"ID":"8be91b06-5f84-4851-ba53-f7ef63f89b53","Type":"ContainerStarted","Data":"442f5727505255198400a28e6d9c7d2091d1003bf863311efc305a6574a1c2dc"} Jul 21 00:24:43.633528 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:43.633489 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:24:43.733869 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:43.733829 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:24:43.913310 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:43.913224 2580 generic.go:356] "Generic (PLEG): container finished" podID="8be91b06-5f84-4851-ba53-f7ef63f89b53" containerID="442f5727505255198400a28e6d9c7d2091d1003bf863311efc305a6574a1c2dc" exitCode=0 Jul 21 00:24:43.913310 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:43.913306 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974" event={"ID":"8be91b06-5f84-4851-ba53-f7ef63f89b53","Type":"ContainerDied","Data":"442f5727505255198400a28e6d9c7d2091d1003bf863311efc305a6574a1c2dc"} Jul 21 00:24:44.919115 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:44.919079 2580 generic.go:356] "Generic (PLEG): container finished" podID="8be91b06-5f84-4851-ba53-f7ef63f89b53" containerID="c7f2ad2410f6b02f85940df49f3b02708957239409fca0315d9f0eadb70bca13" exitCode=0 Jul 21 00:24:44.919553 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:44.919155 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974" event={"ID":"8be91b06-5f84-4851-ba53-f7ef63f89b53","Type":"ContainerDied","Data":"c7f2ad2410f6b02f85940df49f3b02708957239409fca0315d9f0eadb70bca13"} Jul 21 00:24:46.042306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:46.042273 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:24:46.051512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:46.051483 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974" Jul 21 00:24:46.102140 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:46.102105 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:24:46.213406 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:46.213327 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8be91b06-5f84-4851-ba53-f7ef63f89b53-util\") pod \"8be91b06-5f84-4851-ba53-f7ef63f89b53\" (UID: \"8be91b06-5f84-4851-ba53-f7ef63f89b53\") " Jul 21 00:24:46.213406 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:46.213369 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8be91b06-5f84-4851-ba53-f7ef63f89b53-bundle\") pod \"8be91b06-5f84-4851-ba53-f7ef63f89b53\" (UID: \"8be91b06-5f84-4851-ba53-f7ef63f89b53\") " Jul 21 00:24:46.213406 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:46.213392 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5rms\" (UniqueName: \"kubernetes.io/projected/8be91b06-5f84-4851-ba53-f7ef63f89b53-kube-api-access-w5rms\") pod \"8be91b06-5f84-4851-ba53-f7ef63f89b53\" (UID: \"8be91b06-5f84-4851-ba53-f7ef63f89b53\") " Jul 21 00:24:46.213801 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:46.213770 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be91b06-5f84-4851-ba53-f7ef63f89b53-bundle" (OuterVolumeSpecName: "bundle") pod "8be91b06-5f84-4851-ba53-f7ef63f89b53" (UID: "8be91b06-5f84-4851-ba53-f7ef63f89b53"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:24:46.215650 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:46.215629 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be91b06-5f84-4851-ba53-f7ef63f89b53-kube-api-access-w5rms" (OuterVolumeSpecName: "kube-api-access-w5rms") pod "8be91b06-5f84-4851-ba53-f7ef63f89b53" (UID: "8be91b06-5f84-4851-ba53-f7ef63f89b53"). InnerVolumeSpecName "kube-api-access-w5rms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:24:46.217852 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:46.217829 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8be91b06-5f84-4851-ba53-f7ef63f89b53-util" (OuterVolumeSpecName: "util") pod "8be91b06-5f84-4851-ba53-f7ef63f89b53" (UID: "8be91b06-5f84-4851-ba53-f7ef63f89b53"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:24:46.313862 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:46.313818 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8be91b06-5f84-4851-ba53-f7ef63f89b53-util\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:24:46.313862 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:46.313860 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8be91b06-5f84-4851-ba53-f7ef63f89b53-bundle\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:24:46.314050 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:46.313874 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5rms\" (UniqueName: \"kubernetes.io/projected/8be91b06-5f84-4851-ba53-f7ef63f89b53-kube-api-access-w5rms\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:24:46.930623 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:46.930576 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974" Jul 21 00:24:46.930809 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:46.930563 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f7p974" event={"ID":"8be91b06-5f84-4851-ba53-f7ef63f89b53","Type":"ContainerDied","Data":"01bd51324e595a4c456f8e3e62bcb3a791fa847649aa089833ef24239a62c6d2"} Jul 21 00:24:46.930809 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:46.930723 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01bd51324e595a4c456f8e3e62bcb3a791fa847649aa089833ef24239a62c6d2" Jul 21 00:24:49.560252 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:49.560213 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:24:50.235204 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:50.235169 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7rb8f"] Jul 21 00:24:50.235436 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:50.235424 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8be91b06-5f84-4851-ba53-f7ef63f89b53" containerName="extract" Jul 21 00:24:50.235488 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:50.235438 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8be91b06-5f84-4851-ba53-f7ef63f89b53" containerName="extract" Jul 21 00:24:50.235488 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:50.235447 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8be91b06-5f84-4851-ba53-f7ef63f89b53" containerName="util" Jul 21 00:24:50.235488 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:50.235453 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8be91b06-5f84-4851-ba53-f7ef63f89b53" containerName="util" Jul 21 00:24:50.235488 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:50.235467 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8be91b06-5f84-4851-ba53-f7ef63f89b53" containerName="pull" Jul 21 00:24:50.235488 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:50.235473 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8be91b06-5f84-4851-ba53-f7ef63f89b53" containerName="pull" Jul 21 00:24:50.235672 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:50.235514 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8be91b06-5f84-4851-ba53-f7ef63f89b53" containerName="extract" Jul 21 00:24:50.237679 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:50.237663 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-7rb8f" Jul 21 00:24:50.240159 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:50.240136 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-n66fc\"" Jul 21 00:24:50.240287 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:50.240136 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 21 00:24:50.241092 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:50.241077 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 21 00:24:50.246026 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:50.246001 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7rb8f"] Jul 21 00:24:50.346285 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:50.346251 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5dx9\" (UniqueName: \"kubernetes.io/projected/4e35cbac-0f80-4906-bb7c-254c732ae6b6-kube-api-access-x5dx9\") pod \"cert-manager-55b4d954d5-7rb8f\" (UID: \"4e35cbac-0f80-4906-bb7c-254c732ae6b6\") " pod="cert-manager/cert-manager-55b4d954d5-7rb8f" Jul 21 00:24:50.346471 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:50.346293 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e35cbac-0f80-4906-bb7c-254c732ae6b6-bound-sa-token\") pod \"cert-manager-55b4d954d5-7rb8f\" (UID: \"4e35cbac-0f80-4906-bb7c-254c732ae6b6\") " pod="cert-manager/cert-manager-55b4d954d5-7rb8f" Jul 21 00:24:50.446939 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:50.446901 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5dx9\" (UniqueName: \"kubernetes.io/projected/4e35cbac-0f80-4906-bb7c-254c732ae6b6-kube-api-access-x5dx9\") pod \"cert-manager-55b4d954d5-7rb8f\" (UID: \"4e35cbac-0f80-4906-bb7c-254c732ae6b6\") " pod="cert-manager/cert-manager-55b4d954d5-7rb8f" Jul 21 00:24:50.446939 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:50.446944 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e35cbac-0f80-4906-bb7c-254c732ae6b6-bound-sa-token\") pod \"cert-manager-55b4d954d5-7rb8f\" (UID: \"4e35cbac-0f80-4906-bb7c-254c732ae6b6\") " pod="cert-manager/cert-manager-55b4d954d5-7rb8f" Jul 21 00:24:50.455293 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:50.455259 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e35cbac-0f80-4906-bb7c-254c732ae6b6-bound-sa-token\") pod \"cert-manager-55b4d954d5-7rb8f\" (UID: \"4e35cbac-0f80-4906-bb7c-254c732ae6b6\") " pod="cert-manager/cert-manager-55b4d954d5-7rb8f" Jul 21 00:24:50.455293 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:50.455268 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5dx9\" (UniqueName: \"kubernetes.io/projected/4e35cbac-0f80-4906-bb7c-254c732ae6b6-kube-api-access-x5dx9\") pod \"cert-manager-55b4d954d5-7rb8f\" (UID: \"4e35cbac-0f80-4906-bb7c-254c732ae6b6\") " pod="cert-manager/cert-manager-55b4d954d5-7rb8f" Jul 21 00:24:50.547875 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:50.547778 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-7rb8f" Jul 21 00:24:50.681997 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:50.681972 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7rb8f"] Jul 21 00:24:50.684355 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:24:50.684330 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e35cbac_0f80_4906_bb7c_254c732ae6b6.slice/crio-d54b33f71c34ba95cedd21c169e8f5d1266c2bd62fefceafcd427e4e0bf73ae9 WatchSource:0}: Error finding container d54b33f71c34ba95cedd21c169e8f5d1266c2bd62fefceafcd427e4e0bf73ae9: Status 404 returned error can't find the container with id d54b33f71c34ba95cedd21c169e8f5d1266c2bd62fefceafcd427e4e0bf73ae9 Jul 21 00:24:50.952972 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:50.952930 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-7rb8f" event={"ID":"4e35cbac-0f80-4906-bb7c-254c732ae6b6","Type":"ContainerStarted","Data":"d54b33f71c34ba95cedd21c169e8f5d1266c2bd62fefceafcd427e4e0bf73ae9"} Jul 21 00:24:51.338470 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:51.338436 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:24:51.438471 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:51.438430 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:24:53.053704 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:53.053667 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cpgp9"] Jul 21 00:24:53.056205 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:53.056183 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cpgp9" Jul 21 00:24:53.058826 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:53.058807 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 21 00:24:53.059840 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:53.059732 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 21 00:24:53.059840 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:53.059756 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-n7l2z\"" Jul 21 00:24:53.065138 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:53.065086 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cpgp9"] Jul 21 00:24:53.171235 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:53.171197 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb229d2f-6fc3-46a6-a331-cc469cd2587a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cpgp9\" (UID: \"bb229d2f-6fc3-46a6-a331-cc469cd2587a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cpgp9" Jul 21 00:24:53.171437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:53.171254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q429\" (UniqueName: \"kubernetes.io/projected/bb229d2f-6fc3-46a6-a331-cc469cd2587a-kube-api-access-5q429\") pod \"openshift-lws-operator-fd8ccff4c-cpgp9\" (UID: \"bb229d2f-6fc3-46a6-a331-cc469cd2587a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cpgp9" Jul 21 00:24:53.271985 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:53.271940 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb229d2f-6fc3-46a6-a331-cc469cd2587a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cpgp9\" (UID: \"bb229d2f-6fc3-46a6-a331-cc469cd2587a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cpgp9" Jul 21 00:24:53.272201 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:53.272008 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q429\" (UniqueName: \"kubernetes.io/projected/bb229d2f-6fc3-46a6-a331-cc469cd2587a-kube-api-access-5q429\") pod \"openshift-lws-operator-fd8ccff4c-cpgp9\" (UID: \"bb229d2f-6fc3-46a6-a331-cc469cd2587a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cpgp9" Jul 21 00:24:53.272414 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:53.272391 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb229d2f-6fc3-46a6-a331-cc469cd2587a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cpgp9\" (UID: \"bb229d2f-6fc3-46a6-a331-cc469cd2587a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cpgp9" Jul 21 00:24:53.289972 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:53.289935 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q429\" (UniqueName: \"kubernetes.io/projected/bb229d2f-6fc3-46a6-a331-cc469cd2587a-kube-api-access-5q429\") pod \"openshift-lws-operator-fd8ccff4c-cpgp9\" (UID: \"bb229d2f-6fc3-46a6-a331-cc469cd2587a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cpgp9" Jul 21 00:24:53.368560 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:53.368534 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cpgp9" Jul 21 00:24:53.540548 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:53.540525 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cpgp9"] Jul 21 00:24:53.543227 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:24:53.543194 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb229d2f_6fc3_46a6_a331_cc469cd2587a.slice/crio-218ab11e05616a04266819c4c397c2e6ac723d263180c906a290ca935333f7fb WatchSource:0}: Error finding container 218ab11e05616a04266819c4c397c2e6ac723d263180c906a290ca935333f7fb: Status 404 returned error can't find the container with id 218ab11e05616a04266819c4c397c2e6ac723d263180c906a290ca935333f7fb Jul 21 00:24:53.967113 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:53.967081 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-7rb8f" event={"ID":"4e35cbac-0f80-4906-bb7c-254c732ae6b6","Type":"ContainerStarted","Data":"b5a8bcf7d82153559ad7a29390fea8869f8afb7791983d7183baa662adf03f96"} Jul 21 00:24:53.968494 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:53.968457 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cpgp9" event={"ID":"bb229d2f-6fc3-46a6-a331-cc469cd2587a","Type":"ContainerStarted","Data":"218ab11e05616a04266819c4c397c2e6ac723d263180c906a290ca935333f7fb"} Jul 21 00:24:53.988068 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:24:53.988018 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-7rb8f" podStartSLOduration=1.323482806 podStartE2EDuration="3.988000357s" totalImagesPullingTime="2.664517551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:24:53.987499223 +0000 UTC m=+488.202879571" watchObservedRunningTime="2026-07-21 00:24:53.988000357 +0000 UTC m=+488.203380680" Jul 21 00:25:03.008944 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:03.008900 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cpgp9" event={"ID":"bb229d2f-6fc3-46a6-a331-cc469cd2587a","Type":"ContainerStarted","Data":"3ae02a338c13dc1745fa0410db62182df1148dd02b532588d1439b52ce4929fe"} Jul 21 00:25:03.027439 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:03.027373 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cpgp9" podStartSLOduration=1.361661238 podStartE2EDuration="10.02735523s" totalImagesPullingTime="8.665693992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:25:03.024938335 +0000 UTC m=+497.240318661" watchObservedRunningTime="2026-07-21 00:25:03.02735523 +0000 UTC m=+497.242735557" Jul 21 00:25:03.836731 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:03.836698 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:03.937692 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:03.937642 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:05.399908 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:05.399871 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766"] Jul 21 00:25:05.402838 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:05.402815 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766" Jul 21 00:25:05.405382 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:05.405357 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4sl9\"" Jul 21 00:25:05.405382 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:05.405381 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 00:25:05.406149 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:05.406130 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 00:25:05.410672 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:05.410649 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766"] Jul 21 00:25:05.458835 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:05.458798 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b1a46a5-fad8-4eab-be93-9d957b4f2462-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766\" (UID: \"2b1a46a5-fad8-4eab-be93-9d957b4f2462\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766" Jul 21 00:25:05.459014 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:05.458854 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl98m\" (UniqueName: \"kubernetes.io/projected/2b1a46a5-fad8-4eab-be93-9d957b4f2462-kube-api-access-zl98m\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766\" (UID: \"2b1a46a5-fad8-4eab-be93-9d957b4f2462\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766" Jul 21 00:25:05.459014 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:05.458907 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b1a46a5-fad8-4eab-be93-9d957b4f2462-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766\" (UID: \"2b1a46a5-fad8-4eab-be93-9d957b4f2462\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766" Jul 21 00:25:05.559822 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:05.559774 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl98m\" (UniqueName: \"kubernetes.io/projected/2b1a46a5-fad8-4eab-be93-9d957b4f2462-kube-api-access-zl98m\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766\" (UID: \"2b1a46a5-fad8-4eab-be93-9d957b4f2462\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766" Jul 21 00:25:05.559822 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:05.559840 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b1a46a5-fad8-4eab-be93-9d957b4f2462-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766\" (UID: \"2b1a46a5-fad8-4eab-be93-9d957b4f2462\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766" Jul 21 00:25:05.560052 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:05.559868 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b1a46a5-fad8-4eab-be93-9d957b4f2462-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766\" (UID: \"2b1a46a5-fad8-4eab-be93-9d957b4f2462\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766" Jul 21 00:25:05.560263 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:05.560247 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b1a46a5-fad8-4eab-be93-9d957b4f2462-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766\" (UID: \"2b1a46a5-fad8-4eab-be93-9d957b4f2462\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766" Jul 21 00:25:05.560301 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:05.560278 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b1a46a5-fad8-4eab-be93-9d957b4f2462-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766\" (UID: \"2b1a46a5-fad8-4eab-be93-9d957b4f2462\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766" Jul 21 00:25:05.568558 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:05.568513 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl98m\" (UniqueName: \"kubernetes.io/projected/2b1a46a5-fad8-4eab-be93-9d957b4f2462-kube-api-access-zl98m\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766\" (UID: \"2b1a46a5-fad8-4eab-be93-9d957b4f2462\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766" Jul 21 00:25:05.713270 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:05.713178 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766" Jul 21 00:25:05.717747 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:05.717719 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:05.844224 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:05.844190 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766"] Jul 21 00:25:05.847224 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:25:05.847198 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1a46a5_fad8_4eab_be93_9d957b4f2462.slice/crio-f1bb6bb1935dc30a790753ecb71f1f6c32a2347f14c2541bd48606c64bd77d63 WatchSource:0}: Error finding container f1bb6bb1935dc30a790753ecb71f1f6c32a2347f14c2541bd48606c64bd77d63: Status 404 returned error can't find the container with id f1bb6bb1935dc30a790753ecb71f1f6c32a2347f14c2541bd48606c64bd77d63 Jul 21 00:25:06.022658 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:06.022552 2580 generic.go:356] "Generic (PLEG): container finished" podID="2b1a46a5-fad8-4eab-be93-9d957b4f2462" containerID="aa4fece773330654af833cc2cc351397980b044963019a0b58014bf19c9fcf1b" exitCode=0 Jul 21 00:25:06.022658 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:06.022647 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766" event={"ID":"2b1a46a5-fad8-4eab-be93-9d957b4f2462","Type":"ContainerDied","Data":"aa4fece773330654af833cc2cc351397980b044963019a0b58014bf19c9fcf1b"} Jul 21 00:25:06.022858 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:06.022686 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766" event={"ID":"2b1a46a5-fad8-4eab-be93-9d957b4f2462","Type":"ContainerStarted","Data":"f1bb6bb1935dc30a790753ecb71f1f6c32a2347f14c2541bd48606c64bd77d63"} Jul 21 00:25:07.028675 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:07.028565 2580 generic.go:356] "Generic (PLEG): container finished" podID="2b1a46a5-fad8-4eab-be93-9d957b4f2462" containerID="99356bf5f7aa10d23011bbe5df578f5b7b4d22627fb130a9e8cb025698c7cf0d" exitCode=0 Jul 21 00:25:07.028675 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:07.028657 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766" event={"ID":"2b1a46a5-fad8-4eab-be93-9d957b4f2462","Type":"ContainerDied","Data":"99356bf5f7aa10d23011bbe5df578f5b7b4d22627fb130a9e8cb025698c7cf0d"} Jul 21 00:25:08.034262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:08.034227 2580 generic.go:356] "Generic (PLEG): container finished" podID="2b1a46a5-fad8-4eab-be93-9d957b4f2462" containerID="cb566be7044f78c48285735f737ebba39c0c1a01944249430677c20ee0f5252d" exitCode=0 Jul 21 00:25:08.034699 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:08.034315 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766" event={"ID":"2b1a46a5-fad8-4eab-be93-9d957b4f2462","Type":"ContainerDied","Data":"cb566be7044f78c48285735f737ebba39c0c1a01944249430677c20ee0f5252d"} Jul 21 00:25:09.164128 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:09.164101 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766" Jul 21 00:25:09.187654 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:09.187624 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zl98m\" (UniqueName: \"kubernetes.io/projected/2b1a46a5-fad8-4eab-be93-9d957b4f2462-kube-api-access-zl98m\") pod \"2b1a46a5-fad8-4eab-be93-9d957b4f2462\" (UID: \"2b1a46a5-fad8-4eab-be93-9d957b4f2462\") " Jul 21 00:25:09.187839 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:09.187667 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b1a46a5-fad8-4eab-be93-9d957b4f2462-bundle\") pod \"2b1a46a5-fad8-4eab-be93-9d957b4f2462\" (UID: \"2b1a46a5-fad8-4eab-be93-9d957b4f2462\") " Jul 21 00:25:09.187839 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:09.187728 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b1a46a5-fad8-4eab-be93-9d957b4f2462-util\") pod \"2b1a46a5-fad8-4eab-be93-9d957b4f2462\" (UID: \"2b1a46a5-fad8-4eab-be93-9d957b4f2462\") " Jul 21 00:25:09.188655 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:09.188620 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1a46a5-fad8-4eab-be93-9d957b4f2462-bundle" (OuterVolumeSpecName: "bundle") pod "2b1a46a5-fad8-4eab-be93-9d957b4f2462" (UID: "2b1a46a5-fad8-4eab-be93-9d957b4f2462"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:25:09.189963 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:09.189935 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1a46a5-fad8-4eab-be93-9d957b4f2462-kube-api-access-zl98m" (OuterVolumeSpecName: "kube-api-access-zl98m") pod "2b1a46a5-fad8-4eab-be93-9d957b4f2462" (UID: "2b1a46a5-fad8-4eab-be93-9d957b4f2462"). InnerVolumeSpecName "kube-api-access-zl98m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:25:09.195550 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:09.195515 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1a46a5-fad8-4eab-be93-9d957b4f2462-util" (OuterVolumeSpecName: "util") pod "2b1a46a5-fad8-4eab-be93-9d957b4f2462" (UID: "2b1a46a5-fad8-4eab-be93-9d957b4f2462"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:25:09.289052 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:09.288957 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl98m\" (UniqueName: \"kubernetes.io/projected/2b1a46a5-fad8-4eab-be93-9d957b4f2462-kube-api-access-zl98m\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:25:09.289052 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:09.288990 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b1a46a5-fad8-4eab-be93-9d957b4f2462-bundle\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:25:09.289052 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:09.289004 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b1a46a5-fad8-4eab-be93-9d957b4f2462-util\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:25:10.044346 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:10.044318 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766" Jul 21 00:25:10.044531 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:10.044318 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af9s766" event={"ID":"2b1a46a5-fad8-4eab-be93-9d957b4f2462","Type":"ContainerDied","Data":"f1bb6bb1935dc30a790753ecb71f1f6c32a2347f14c2541bd48606c64bd77d63"} Jul 21 00:25:10.044531 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:10.044436 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1bb6bb1935dc30a790753ecb71f1f6c32a2347f14c2541bd48606c64bd77d63" Jul 21 00:25:14.056239 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:14.056206 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:14.154859 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:14.154823 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:18.471132 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:18.471087 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:18.569000 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:18.568962 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:19.089856 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.089813 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:19.398106 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.398014 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:19.500866 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.500830 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:19.515529 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.515491 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7"] Jul 21 00:25:19.515874 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.515856 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b1a46a5-fad8-4eab-be93-9d957b4f2462" containerName="extract" Jul 21 00:25:19.515960 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.515877 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1a46a5-fad8-4eab-be93-9d957b4f2462" containerName="extract" Jul 21 00:25:19.515960 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.515895 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b1a46a5-fad8-4eab-be93-9d957b4f2462" containerName="pull" Jul 21 00:25:19.515960 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.515902 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1a46a5-fad8-4eab-be93-9d957b4f2462" containerName="pull" Jul 21 00:25:19.515960 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.515917 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b1a46a5-fad8-4eab-be93-9d957b4f2462" containerName="util" Jul 21 00:25:19.515960 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.515926 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1a46a5-fad8-4eab-be93-9d957b4f2462" containerName="util" Jul 21 00:25:19.516184 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.515999 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b1a46a5-fad8-4eab-be93-9d957b4f2462" containerName="extract" Jul 21 00:25:19.520822 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.520796 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7" Jul 21 00:25:19.523543 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.523522 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7h555\"" Jul 21 00:25:19.523677 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.523522 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 21 00:25:19.523677 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.523522 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 21 00:25:19.523896 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.523880 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 21 00:25:19.527954 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.527934 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 21 00:25:19.540184 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.540147 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7"] Jul 21 00:25:19.572206 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.572165 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/49be6af6-766d-431d-b0cd-0963a5695e30-manifests\") pod \"opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7\" (UID: \"49be6af6-766d-431d-b0cd-0963a5695e30\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7" Jul 21 00:25:19.572360 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.572213 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49be6af6-766d-431d-b0cd-0963a5695e30-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7\" (UID: \"49be6af6-766d-431d-b0cd-0963a5695e30\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7" Jul 21 00:25:19.572360 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.572243 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q8pt\" (UniqueName: \"kubernetes.io/projected/49be6af6-766d-431d-b0cd-0963a5695e30-kube-api-access-2q8pt\") pod \"opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7\" (UID: \"49be6af6-766d-431d-b0cd-0963a5695e30\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7" Jul 21 00:25:19.572360 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.572303 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49be6af6-766d-431d-b0cd-0963a5695e30-tmp\") pod \"opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7\" (UID: \"49be6af6-766d-431d-b0cd-0963a5695e30\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7" Jul 21 00:25:19.572360 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.572329 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/49be6af6-766d-431d-b0cd-0963a5695e30-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7\" (UID: \"49be6af6-766d-431d-b0cd-0963a5695e30\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7" Jul 21 00:25:19.673660 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.673537 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49be6af6-766d-431d-b0cd-0963a5695e30-tmp\") pod \"opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7\" (UID: \"49be6af6-766d-431d-b0cd-0963a5695e30\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7" Jul 21 00:25:19.673660 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.673594 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/49be6af6-766d-431d-b0cd-0963a5695e30-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7\" (UID: \"49be6af6-766d-431d-b0cd-0963a5695e30\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7" Jul 21 00:25:19.673838 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.673674 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/49be6af6-766d-431d-b0cd-0963a5695e30-manifests\") pod \"opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7\" (UID: \"49be6af6-766d-431d-b0cd-0963a5695e30\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7" Jul 21 00:25:19.673838 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.673713 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49be6af6-766d-431d-b0cd-0963a5695e30-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7\" (UID: \"49be6af6-766d-431d-b0cd-0963a5695e30\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7" Jul 21 00:25:19.673838 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.673738 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q8pt\" (UniqueName: \"kubernetes.io/projected/49be6af6-766d-431d-b0cd-0963a5695e30-kube-api-access-2q8pt\") pod \"opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7\" (UID: \"49be6af6-766d-431d-b0cd-0963a5695e30\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7" Jul 21 00:25:19.674137 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.674102 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/49be6af6-766d-431d-b0cd-0963a5695e30-manifests\") pod \"opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7\" (UID: \"49be6af6-766d-431d-b0cd-0963a5695e30\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7" Jul 21 00:25:19.676028 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.675994 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49be6af6-766d-431d-b0cd-0963a5695e30-tmp\") pod \"opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7\" (UID: \"49be6af6-766d-431d-b0cd-0963a5695e30\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7" Jul 21 00:25:19.676291 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.676271 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/49be6af6-766d-431d-b0cd-0963a5695e30-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7\" (UID: \"49be6af6-766d-431d-b0cd-0963a5695e30\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7" Jul 21 00:25:19.676334 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.676280 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49be6af6-766d-431d-b0cd-0963a5695e30-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7\" (UID: \"49be6af6-766d-431d-b0cd-0963a5695e30\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7" Jul 21 00:25:19.684785 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.684757 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q8pt\" (UniqueName: \"kubernetes.io/projected/49be6af6-766d-431d-b0cd-0963a5695e30-kube-api-access-2q8pt\") pod \"opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7\" (UID: \"49be6af6-766d-431d-b0cd-0963a5695e30\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7" Jul 21 00:25:19.769824 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.769791 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:19.830852 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.830814 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7" Jul 21 00:25:19.912137 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.912096 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:19.972850 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.972823 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7"] Jul 21 00:25:19.975404 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:25:19.975378 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49be6af6_766d_431d_b0cd_0963a5695e30.slice/crio-560e0d8d5a5e95ee87cc9fe07d90d81f8321d8ed270a126ba4d0ee14836e8792 WatchSource:0}: Error finding container 560e0d8d5a5e95ee87cc9fe07d90d81f8321d8ed270a126ba4d0ee14836e8792: Status 404 returned error can't find the container with id 560e0d8d5a5e95ee87cc9fe07d90d81f8321d8ed270a126ba4d0ee14836e8792 Jul 21 00:25:19.993216 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:19.993182 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:20.062206 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:20.062175 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:20.085980 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:20.085948 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7" event={"ID":"49be6af6-766d-431d-b0cd-0963a5695e30","Type":"ContainerStarted","Data":"560e0d8d5a5e95ee87cc9fe07d90d81f8321d8ed270a126ba4d0ee14836e8792"} Jul 21 00:25:24.109233 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:24.109191 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7" event={"ID":"49be6af6-766d-431d-b0cd-0963a5695e30","Type":"ContainerStarted","Data":"268a400ae02747e9e0f0d171ce2d455ea3e644062f752ae75081527df68b94cd"} Jul 21 00:25:25.114592 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:25.114550 2580 generic.go:356] "Generic (PLEG): container finished" podID="49be6af6-766d-431d-b0cd-0963a5695e30" containerID="268a400ae02747e9e0f0d171ce2d455ea3e644062f752ae75081527df68b94cd" exitCode=0 Jul 21 00:25:25.115023 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:25.114635 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7" event={"ID":"49be6af6-766d-431d-b0cd-0963a5695e30","Type":"ContainerDied","Data":"268a400ae02747e9e0f0d171ce2d455ea3e644062f752ae75081527df68b94cd"} Jul 21 00:25:26.121456 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:26.121417 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7" event={"ID":"49be6af6-766d-431d-b0cd-0963a5695e30","Type":"ContainerStarted","Data":"49b328ffefddb9edfa074a7070152f3cd5bc906bdb3a9d3ad29504f80293fd89"} Jul 21 00:25:26.121929 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:26.121511 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7" Jul 21 00:25:26.142641 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:26.142569 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7" podStartSLOduration=2.439056403 podStartE2EDuration="7.14254985s" totalImagesPullingTime="3.620324389s" totalInitContainerRuntime="1.083169058s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:25:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:25:26.140691088 +0000 UTC m=+520.356071413" watchObservedRunningTime="2026-07-21 00:25:26.14254985 +0000 UTC m=+520.357930174" Jul 21 00:25:37.129032 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:37.128957 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7" Jul 21 00:25:38.709995 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:38.709944 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:38.889667 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:38.889629 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:39.435433 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.435394 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:39.470751 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.470712 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-66766cb76c-plvdj"] Jul 21 00:25:39.473241 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.473224 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66766cb76c-plvdj" Jul 21 00:25:39.476153 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.476123 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 21 00:25:39.476292 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.476245 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-nnt5c\"" Jul 21 00:25:39.477134 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.477118 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 21 00:25:39.477200 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.477181 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 21 00:25:39.477246 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.477203 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 21 00:25:39.494586 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.494553 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66766cb76c-plvdj"] Jul 21 00:25:39.638052 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.638008 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf25e5c5-65e1-472c-b2d0-0623a79a7e4e-tls-certs\") pod \"kube-auth-proxy-66766cb76c-plvdj\" (UID: \"bf25e5c5-65e1-472c-b2d0-0623a79a7e4e\") " pod="openshift-ingress/kube-auth-proxy-66766cb76c-plvdj" Jul 21 00:25:39.638211 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.638076 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf25e5c5-65e1-472c-b2d0-0623a79a7e4e-tmp\") pod \"kube-auth-proxy-66766cb76c-plvdj\" (UID: \"bf25e5c5-65e1-472c-b2d0-0623a79a7e4e\") " pod="openshift-ingress/kube-auth-proxy-66766cb76c-plvdj" Jul 21 00:25:39.638211 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.638151 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4drrn\" (UniqueName: \"kubernetes.io/projected/bf25e5c5-65e1-472c-b2d0-0623a79a7e4e-kube-api-access-4drrn\") pod \"kube-auth-proxy-66766cb76c-plvdj\" (UID: \"bf25e5c5-65e1-472c-b2d0-0623a79a7e4e\") " pod="openshift-ingress/kube-auth-proxy-66766cb76c-plvdj" Jul 21 00:25:39.712913 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.712828 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-h24w6"] Jul 21 00:25:39.715151 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.715128 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-h24w6" Jul 21 00:25:39.717752 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.717730 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 21 00:25:39.717869 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.717732 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-zbwqq\"" Jul 21 00:25:39.733673 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.733641 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-h24w6"] Jul 21 00:25:39.738858 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.738829 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf25e5c5-65e1-472c-b2d0-0623a79a7e4e-tmp\") pod \"kube-auth-proxy-66766cb76c-plvdj\" (UID: \"bf25e5c5-65e1-472c-b2d0-0623a79a7e4e\") " pod="openshift-ingress/kube-auth-proxy-66766cb76c-plvdj" Jul 21 00:25:39.739037 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.738889 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4drrn\" (UniqueName: \"kubernetes.io/projected/bf25e5c5-65e1-472c-b2d0-0623a79a7e4e-kube-api-access-4drrn\") pod \"kube-auth-proxy-66766cb76c-plvdj\" (UID: \"bf25e5c5-65e1-472c-b2d0-0623a79a7e4e\") " pod="openshift-ingress/kube-auth-proxy-66766cb76c-plvdj" Jul 21 00:25:39.739037 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.738909 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf25e5c5-65e1-472c-b2d0-0623a79a7e4e-tls-certs\") pod \"kube-auth-proxy-66766cb76c-plvdj\" (UID: \"bf25e5c5-65e1-472c-b2d0-0623a79a7e4e\") " pod="openshift-ingress/kube-auth-proxy-66766cb76c-plvdj" Jul 21 00:25:39.739146 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:25:39.739039 2580 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 21 00:25:39.739146 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:25:39.739121 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf25e5c5-65e1-472c-b2d0-0623a79a7e4e-tls-certs podName:bf25e5c5-65e1-472c-b2d0-0623a79a7e4e nodeName:}" failed. No retries permitted until 2026-07-21 00:25:40.239099572 +0000 UTC m=+534.454479875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/bf25e5c5-65e1-472c-b2d0-0623a79a7e4e-tls-certs") pod "kube-auth-proxy-66766cb76c-plvdj" (UID: "bf25e5c5-65e1-472c-b2d0-0623a79a7e4e") : secret "kube-auth-proxy-tls" not found Jul 21 00:25:39.741252 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.741227 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf25e5c5-65e1-472c-b2d0-0623a79a7e4e-tmp\") pod \"kube-auth-proxy-66766cb76c-plvdj\" (UID: \"bf25e5c5-65e1-472c-b2d0-0623a79a7e4e\") " pod="openshift-ingress/kube-auth-proxy-66766cb76c-plvdj" Jul 21 00:25:39.750935 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.750907 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4drrn\" (UniqueName: \"kubernetes.io/projected/bf25e5c5-65e1-472c-b2d0-0623a79a7e4e-kube-api-access-4drrn\") pod \"kube-auth-proxy-66766cb76c-plvdj\" (UID: \"bf25e5c5-65e1-472c-b2d0-0623a79a7e4e\") " pod="openshift-ingress/kube-auth-proxy-66766cb76c-plvdj" Jul 21 00:25:39.766662 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.766620 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:39.839617 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.839572 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46g86\" (UniqueName: \"kubernetes.io/projected/8ce97c9b-8333-4464-a317-33fec0ef2813-kube-api-access-46g86\") pod \"odh-model-controller-7998c67dc4-h24w6\" (UID: \"8ce97c9b-8333-4464-a317-33fec0ef2813\") " pod="opendatahub/odh-model-controller-7998c67dc4-h24w6" Jul 21 00:25:39.839800 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.839737 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ce97c9b-8333-4464-a317-33fec0ef2813-cert\") pod \"odh-model-controller-7998c67dc4-h24w6\" (UID: \"8ce97c9b-8333-4464-a317-33fec0ef2813\") " pod="opendatahub/odh-model-controller-7998c67dc4-h24w6" Jul 21 00:25:39.902448 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.902416 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:39.940698 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.940658 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ce97c9b-8333-4464-a317-33fec0ef2813-cert\") pod \"odh-model-controller-7998c67dc4-h24w6\" (UID: \"8ce97c9b-8333-4464-a317-33fec0ef2813\") " pod="opendatahub/odh-model-controller-7998c67dc4-h24w6" Jul 21 00:25:39.940877 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.940734 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46g86\" (UniqueName: \"kubernetes.io/projected/8ce97c9b-8333-4464-a317-33fec0ef2813-kube-api-access-46g86\") pod \"odh-model-controller-7998c67dc4-h24w6\" (UID: \"8ce97c9b-8333-4464-a317-33fec0ef2813\") " pod="opendatahub/odh-model-controller-7998c67dc4-h24w6" Jul 21 00:25:39.940877 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:25:39.940813 2580 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 21 00:25:39.940877 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:25:39.940875 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ce97c9b-8333-4464-a317-33fec0ef2813-cert podName:8ce97c9b-8333-4464-a317-33fec0ef2813 nodeName:}" failed. No retries permitted until 2026-07-21 00:25:40.44086053 +0000 UTC m=+534.656240834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8ce97c9b-8333-4464-a317-33fec0ef2813-cert") pod "odh-model-controller-7998c67dc4-h24w6" (UID: "8ce97c9b-8333-4464-a317-33fec0ef2813") : secret "odh-model-controller-webhook-cert" not found Jul 21 00:25:39.951339 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:39.951309 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46g86\" (UniqueName: \"kubernetes.io/projected/8ce97c9b-8333-4464-a317-33fec0ef2813-kube-api-access-46g86\") pod \"odh-model-controller-7998c67dc4-h24w6\" (UID: \"8ce97c9b-8333-4464-a317-33fec0ef2813\") " pod="opendatahub/odh-model-controller-7998c67dc4-h24w6" Jul 21 00:25:40.008532 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.008417 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:40.242748 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.242705 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf25e5c5-65e1-472c-b2d0-0623a79a7e4e-tls-certs\") pod \"kube-auth-proxy-66766cb76c-plvdj\" (UID: \"bf25e5c5-65e1-472c-b2d0-0623a79a7e4e\") " pod="openshift-ingress/kube-auth-proxy-66766cb76c-plvdj" Jul 21 00:25:40.245750 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.245725 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf25e5c5-65e1-472c-b2d0-0623a79a7e4e-tls-certs\") pod \"kube-auth-proxy-66766cb76c-plvdj\" (UID: \"bf25e5c5-65e1-472c-b2d0-0623a79a7e4e\") " pod="openshift-ingress/kube-auth-proxy-66766cb76c-plvdj" Jul 21 00:25:40.253080 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.253035 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:40.383167 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.383132 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66766cb76c-plvdj" Jul 21 00:25:40.444661 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.444625 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ce97c9b-8333-4464-a317-33fec0ef2813-cert\") pod \"odh-model-controller-7998c67dc4-h24w6\" (UID: \"8ce97c9b-8333-4464-a317-33fec0ef2813\") " pod="opendatahub/odh-model-controller-7998c67dc4-h24w6" Jul 21 00:25:40.447536 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.447508 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ce97c9b-8333-4464-a317-33fec0ef2813-cert\") pod \"odh-model-controller-7998c67dc4-h24w6\" (UID: \"8ce97c9b-8333-4464-a317-33fec0ef2813\") " pod="opendatahub/odh-model-controller-7998c67dc4-h24w6" Jul 21 00:25:40.519864 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.519839 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66766cb76c-plvdj"] Jul 21 00:25:40.522301 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:25:40.522270 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf25e5c5_65e1_472c_b2d0_0623a79a7e4e.slice/crio-8a83a9ad02916d70519e8118f51ae086e28a6c40aa9acabf93dd31fed582a5f8 WatchSource:0}: Error finding container 8a83a9ad02916d70519e8118f51ae086e28a6c40aa9acabf93dd31fed582a5f8: Status 404 returned error can't find the container with id 8a83a9ad02916d70519e8118f51ae086e28a6c40aa9acabf93dd31fed582a5f8 Jul 21 00:25:40.621638 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.621571 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2tln2"] Jul 21 00:25:40.624993 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.624962 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-h24w6" Jul 21 00:25:40.625160 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.625004 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-2tln2" Jul 21 00:25:40.627417 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.627399 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 21 00:25:40.640019 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.639989 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2tln2"] Jul 21 00:25:40.655464 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.655428 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:40.748384 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.748343 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a39002d-d7f1-4023-a125-2832e32db4f7-tls-certs\") pod \"model-serving-api-54d89bd5fc-2tln2\" (UID: \"6a39002d-d7f1-4023-a125-2832e32db4f7\") " pod="opendatahub/model-serving-api-54d89bd5fc-2tln2" Jul 21 00:25:40.748825 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.748419 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgfhq\" (UniqueName: \"kubernetes.io/projected/6a39002d-d7f1-4023-a125-2832e32db4f7-kube-api-access-fgfhq\") pod \"model-serving-api-54d89bd5fc-2tln2\" (UID: \"6a39002d-d7f1-4023-a125-2832e32db4f7\") " pod="opendatahub/model-serving-api-54d89bd5fc-2tln2" Jul 21 00:25:40.754460 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.754431 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dknzz"] Jul 21 00:25:40.757564 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.757547 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:40.760388 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.760198 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 21 00:25:40.760388 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.760198 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 21 00:25:40.760388 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.760278 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rnl52\"" Jul 21 00:25:40.760711 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.760654 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 21 00:25:40.760711 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.760678 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 21 00:25:40.761592 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.761574 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-h24w6"] Jul 21 00:25:40.764453 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:25:40.764427 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce97c9b_8333_4464_a317_33fec0ef2813.slice/crio-4033c46684e362613a3cf92b2a7ebec525a936eb9aee0b498555b201b6ec1d21 WatchSource:0}: Error finding container 4033c46684e362613a3cf92b2a7ebec525a936eb9aee0b498555b201b6ec1d21: Status 404 returned error can't find the container with id 4033c46684e362613a3cf92b2a7ebec525a936eb9aee0b498555b201b6ec1d21 Jul 21 00:25:40.769671 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.769635 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:40.770456 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.770437 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dknzz"] Jul 21 00:25:40.849459 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.849422 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-local-certs\") pod \"istiod-openshift-gateway-7574459656-dknzz\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:40.849459 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.849461 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-dknzz\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:40.849709 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.849500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a39002d-d7f1-4023-a125-2832e32db4f7-tls-certs\") pod \"model-serving-api-54d89bd5fc-2tln2\" (UID: \"6a39002d-d7f1-4023-a125-2832e32db4f7\") " pod="opendatahub/model-serving-api-54d89bd5fc-2tln2" Jul 21 00:25:40.849709 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.849563 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-istio-token\") pod \"istiod-openshift-gateway-7574459656-dknzz\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:40.849709 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.849650 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgfhq\" (UniqueName: \"kubernetes.io/projected/6a39002d-d7f1-4023-a125-2832e32db4f7-kube-api-access-fgfhq\") pod \"model-serving-api-54d89bd5fc-2tln2\" (UID: \"6a39002d-d7f1-4023-a125-2832e32db4f7\") " pod="opendatahub/model-serving-api-54d89bd5fc-2tln2" Jul 21 00:25:40.849709 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.849692 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2wpp\" (UniqueName: \"kubernetes.io/projected/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-kube-api-access-l2wpp\") pod \"istiod-openshift-gateway-7574459656-dknzz\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:40.849847 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.849718 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-cacerts\") pod \"istiod-openshift-gateway-7574459656-dknzz\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:40.849847 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.849766 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-dknzz\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:40.849847 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.849787 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-dknzz\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:40.852036 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.852014 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a39002d-d7f1-4023-a125-2832e32db4f7-tls-certs\") pod \"model-serving-api-54d89bd5fc-2tln2\" (UID: \"6a39002d-d7f1-4023-a125-2832e32db4f7\") " pod="opendatahub/model-serving-api-54d89bd5fc-2tln2" Jul 21 00:25:40.859454 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.859427 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgfhq\" (UniqueName: \"kubernetes.io/projected/6a39002d-d7f1-4023-a125-2832e32db4f7-kube-api-access-fgfhq\") pod \"model-serving-api-54d89bd5fc-2tln2\" (UID: \"6a39002d-d7f1-4023-a125-2832e32db4f7\") " pod="opendatahub/model-serving-api-54d89bd5fc-2tln2" Jul 21 00:25:40.905724 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.905619 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:40.935469 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.935427 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-2tln2" Jul 21 00:25:40.951182 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.951150 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-istio-token\") pod \"istiod-openshift-gateway-7574459656-dknzz\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:40.951342 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.951243 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2wpp\" (UniqueName: \"kubernetes.io/projected/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-kube-api-access-l2wpp\") pod \"istiod-openshift-gateway-7574459656-dknzz\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:40.951342 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.951280 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-cacerts\") pod \"istiod-openshift-gateway-7574459656-dknzz\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:40.951342 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.951311 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-dknzz\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:40.951342 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.951338 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-dknzz\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:40.951543 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.951368 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-local-certs\") pod \"istiod-openshift-gateway-7574459656-dknzz\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:40.951543 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.951401 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-dknzz\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:40.952088 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.952061 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-dknzz\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:40.953933 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.953912 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-dknzz\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:40.954301 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.954281 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-local-certs\") pod \"istiod-openshift-gateway-7574459656-dknzz\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:40.954391 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.954336 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-dknzz\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:40.954515 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.954489 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-cacerts\") pod \"istiod-openshift-gateway-7574459656-dknzz\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:40.960250 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.960227 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-istio-token\") pod \"istiod-openshift-gateway-7574459656-dknzz\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:40.960250 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:40.960240 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2wpp\" (UniqueName: \"kubernetes.io/projected/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-kube-api-access-l2wpp\") pod \"istiod-openshift-gateway-7574459656-dknzz\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:41.007124 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:41.007076 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:41.073556 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:41.073412 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2tln2"] Jul 21 00:25:41.075993 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:41.075967 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:41.076231 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:25:41.076211 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a39002d_d7f1_4023_a125_2832e32db4f7.slice/crio-f2f4e969784bda07a2cdb7c4615e78f28b3831418f854f4e57509cfd33322fb0 WatchSource:0}: Error finding container f2f4e969784bda07a2cdb7c4615e78f28b3831418f854f4e57509cfd33322fb0: Status 404 returned error can't find the container with id f2f4e969784bda07a2cdb7c4615e78f28b3831418f854f4e57509cfd33322fb0 Jul 21 00:25:41.189540 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:41.189497 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-h24w6" event={"ID":"8ce97c9b-8333-4464-a317-33fec0ef2813","Type":"ContainerStarted","Data":"4033c46684e362613a3cf92b2a7ebec525a936eb9aee0b498555b201b6ec1d21"} Jul 21 00:25:41.190958 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:41.190933 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66766cb76c-plvdj" event={"ID":"bf25e5c5-65e1-472c-b2d0-0623a79a7e4e","Type":"ContainerStarted","Data":"8a83a9ad02916d70519e8118f51ae086e28a6c40aa9acabf93dd31fed582a5f8"} Jul 21 00:25:41.192084 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:41.192055 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-2tln2" event={"ID":"6a39002d-d7f1-4023-a125-2832e32db4f7","Type":"ContainerStarted","Data":"f2f4e969784bda07a2cdb7c4615e78f28b3831418f854f4e57509cfd33322fb0"} Jul 21 00:25:41.215803 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:41.215769 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dknzz"] Jul 21 00:25:41.218986 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:25:41.218960 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode51af2db_d5f5_4d52_954d_1fcdbb1b3454.slice/crio-57ec430d540076ae0d2c6cdcb241c63e3a05b46cb07c92eac5de6acec7f33a91 WatchSource:0}: Error finding container 57ec430d540076ae0d2c6cdcb241c63e3a05b46cb07c92eac5de6acec7f33a91: Status 404 returned error can't find the container with id 57ec430d540076ae0d2c6cdcb241c63e3a05b46cb07c92eac5de6acec7f33a91 Jul 21 00:25:41.674407 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:41.674367 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:42.204175 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:42.204088 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" event={"ID":"e51af2db-d5f5-4d52-954d-1fcdbb1b3454","Type":"ContainerStarted","Data":"57ec430d540076ae0d2c6cdcb241c63e3a05b46cb07c92eac5de6acec7f33a91"} Jul 21 00:25:45.746710 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:45.746672 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:25:45.746996 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:45.746767 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:25:46.225245 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:46.225192 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" event={"ID":"e51af2db-d5f5-4d52-954d-1fcdbb1b3454","Type":"ContainerStarted","Data":"214b64a29ec9f0aff4290a65f66e4d2342c0d5e434561ee437222d59c55955ef"} Jul 21 00:25:46.225447 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:46.225294 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:46.226833 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:46.226801 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-h24w6" event={"ID":"8ce97c9b-8333-4464-a317-33fec0ef2813","Type":"ContainerStarted","Data":"65ed92bd0251a578cd16fcb44dd5b94fb312e0977cd66925020d7b323eabaf8e"} Jul 21 00:25:46.226972 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:46.226875 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-h24w6" Jul 21 00:25:46.227441 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:46.227419 2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-dknzz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 00:25:46.227560 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:46.227460 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" podUID="e51af2db-d5f5-4d52-954d-1fcdbb1b3454" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 00:25:46.228547 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:46.228527 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66766cb76c-plvdj" event={"ID":"bf25e5c5-65e1-472c-b2d0-0623a79a7e4e","Type":"ContainerStarted","Data":"77dca7cac049ed6c1967dce28cbf82f0070571eaedf20d499a9ce73f4a650128"} Jul 21 00:25:46.230009 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:46.229987 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-2tln2" event={"ID":"6a39002d-d7f1-4023-a125-2832e32db4f7","Type":"ContainerStarted","Data":"dc8487c389a4dadd2ecd617972b9fb1e566fa3466e59c27143fa3d4744bdcb1f"} Jul 21 00:25:46.230115 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:46.230100 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-2tln2" Jul 21 00:25:46.254212 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:46.254153 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" podStartSLOduration=1.7288237560000002 podStartE2EDuration="6.254132772s" totalImagesPullingTime="4.525309016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:25:46.251899201 +0000 UTC m=+540.467279528" watchObservedRunningTime="2026-07-21 00:25:46.254132772 +0000 UTC m=+540.469513098" Jul 21 00:25:46.290004 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:46.289950 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-2tln2" podStartSLOduration=1.628402772 podStartE2EDuration="6.289932723s" totalImagesPullingTime="4.661529951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:25:46.289265881 +0000 UTC m=+540.504646207" watchObservedRunningTime="2026-07-21 00:25:46.289932723 +0000 UTC m=+540.505313057" Jul 21 00:25:46.332378 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:46.332325 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-h24w6" podStartSLOduration=2.358326848 podStartE2EDuration="7.332308657s" totalImagesPullingTime="4.973981809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:25:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:25:46.332245693 +0000 UTC m=+540.547626030" watchObservedRunningTime="2026-07-21 00:25:46.332308657 +0000 UTC m=+540.547689019" Jul 21 00:25:46.375730 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:46.375675 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-66766cb76c-plvdj" podStartSLOduration=2.154163039 podStartE2EDuration="7.375658052s" totalImagesPullingTime="5.221495013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:25:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:25:46.375486999 +0000 UTC m=+540.590867323" watchObservedRunningTime="2026-07-21 00:25:46.375658052 +0000 UTC m=+540.591038375" Jul 21 00:25:46.720576 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:46.720531 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:46.827921 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:46.827889 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:47.235942 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.235907 2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-dknzz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 00:25:47.236108 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.235955 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" podUID="e51af2db-d5f5-4d52-954d-1fcdbb1b3454" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 00:25:47.484717 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.484678 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx"] Jul 21 00:25:47.488335 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.488290 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.491135 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.491118 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-57hxf\"" Jul 21 00:25:47.506743 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.506710 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx"] Jul 21 00:25:47.522501 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.522472 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:47.612765 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.612734 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d01be36-93d9-4180-90cf-ca8f7fde13b9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.612939 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.612779 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d01be36-93d9-4180-90cf-ca8f7fde13b9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.612939 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.612879 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d01be36-93d9-4180-90cf-ca8f7fde13b9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.612939 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.612912 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d01be36-93d9-4180-90cf-ca8f7fde13b9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.613043 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.612957 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d01be36-93d9-4180-90cf-ca8f7fde13b9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.613043 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.612983 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d01be36-93d9-4180-90cf-ca8f7fde13b9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.613043 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.613020 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d01be36-93d9-4180-90cf-ca8f7fde13b9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.613129 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.613052 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d01be36-93d9-4180-90cf-ca8f7fde13b9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.613129 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.613077 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5fbv\" (UniqueName: \"kubernetes.io/projected/4d01be36-93d9-4180-90cf-ca8f7fde13b9-kube-api-access-q5fbv\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.638910 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.638876 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:47.714100 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.714061 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5fbv\" (UniqueName: \"kubernetes.io/projected/4d01be36-93d9-4180-90cf-ca8f7fde13b9-kube-api-access-q5fbv\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.714100 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.714103 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d01be36-93d9-4180-90cf-ca8f7fde13b9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.714346 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.714133 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d01be36-93d9-4180-90cf-ca8f7fde13b9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.714346 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.714164 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d01be36-93d9-4180-90cf-ca8f7fde13b9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.714346 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.714182 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d01be36-93d9-4180-90cf-ca8f7fde13b9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.714346 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.714208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d01be36-93d9-4180-90cf-ca8f7fde13b9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.714346 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.714223 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d01be36-93d9-4180-90cf-ca8f7fde13b9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.714346 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.714253 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d01be36-93d9-4180-90cf-ca8f7fde13b9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.714346 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.714294 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d01be36-93d9-4180-90cf-ca8f7fde13b9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.714728 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.714695 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d01be36-93d9-4180-90cf-ca8f7fde13b9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.714859 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.714788 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d01be36-93d9-4180-90cf-ca8f7fde13b9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.714859 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.714826 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d01be36-93d9-4180-90cf-ca8f7fde13b9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.714965 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.714948 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d01be36-93d9-4180-90cf-ca8f7fde13b9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.715022 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.714977 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d01be36-93d9-4180-90cf-ca8f7fde13b9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.716884 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.716858 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d01be36-93d9-4180-90cf-ca8f7fde13b9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.717116 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.717098 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d01be36-93d9-4180-90cf-ca8f7fde13b9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.727260 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.727232 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d01be36-93d9-4180-90cf-ca8f7fde13b9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.729394 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.729370 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5fbv\" (UniqueName: \"kubernetes.io/projected/4d01be36-93d9-4180-90cf-ca8f7fde13b9-kube-api-access-q5fbv\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4nzxzx\" (UID: \"4d01be36-93d9-4180-90cf-ca8f7fde13b9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.798589 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.798494 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:47.919530 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.919495 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:47.947276 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:47.947252 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx"] Jul 21 00:25:47.949829 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:25:47.949798 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d01be36_93d9_4180_90cf_ca8f7fde13b9.slice/crio-7f92de8e0321b7ebee3028a9c1d51c466be7cf2737cecdf678017abc646ce366 WatchSource:0}: Error finding container 7f92de8e0321b7ebee3028a9c1d51c466be7cf2737cecdf678017abc646ce366: Status 404 returned error can't find the container with id 7f92de8e0321b7ebee3028a9c1d51c466be7cf2737cecdf678017abc646ce366 Jul 21 00:25:48.240676 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:48.240636 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" event={"ID":"4d01be36-93d9-4180-90cf-ca8f7fde13b9","Type":"ContainerStarted","Data":"7f92de8e0321b7ebee3028a9c1d51c466be7cf2737cecdf678017abc646ce366"} Jul 21 00:25:50.043714 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.043653 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:50.142516 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.117304 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xpb5v"] Jul 21 00:25:50.142516 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.128652 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-s6fs8"] Jul 21 00:25:50.142516 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.132091 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xpb5v" Jul 21 00:25:50.142516 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.141806 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85364ca8-8d29-4a95-a39c-b9ea72ab6873-cert\") pod \"kserve-controller-manager-d6f4b65b8-xpb5v\" (UID: \"85364ca8-8d29-4a95-a39c-b9ea72ab6873\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xpb5v" Jul 21 00:25:50.142516 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.141852 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsn5t\" (UniqueName: \"kubernetes.io/projected/85364ca8-8d29-4a95-a39c-b9ea72ab6873-kube-api-access-wsn5t\") pod \"kserve-controller-manager-d6f4b65b8-xpb5v\" (UID: \"85364ca8-8d29-4a95-a39c-b9ea72ab6873\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xpb5v" Jul 21 00:25:50.142516 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.142124 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s6fs8" Jul 21 00:25:50.144660 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.144633 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 21 00:25:50.144811 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.144676 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-cb8qr\"" Jul 21 00:25:50.144811 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.144731 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-88pv6\"" Jul 21 00:25:50.144811 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.144633 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 21 00:25:50.144947 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.144890 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xpb5v"] Jul 21 00:25:50.163327 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.163290 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:50.165759 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.165731 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-s6fs8"] Jul 21 00:25:50.236163 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.236124 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:50.242262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.242221 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qjgs\" (UniqueName: \"kubernetes.io/projected/01b985e5-79f1-4822-b3d5-93bf24f5599e-kube-api-access-7qjgs\") pod \"llmisvc-controller-manager-777c6bd7d5-s6fs8\" (UID: \"01b985e5-79f1-4822-b3d5-93bf24f5599e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s6fs8" Jul 21 00:25:50.242424 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.242395 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01b985e5-79f1-4822-b3d5-93bf24f5599e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-s6fs8\" (UID: \"01b985e5-79f1-4822-b3d5-93bf24f5599e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s6fs8" Jul 21 00:25:50.242496 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.242481 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85364ca8-8d29-4a95-a39c-b9ea72ab6873-cert\") pod \"kserve-controller-manager-d6f4b65b8-xpb5v\" (UID: \"85364ca8-8d29-4a95-a39c-b9ea72ab6873\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xpb5v" Jul 21 00:25:50.242636 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.242613 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsn5t\" (UniqueName: \"kubernetes.io/projected/85364ca8-8d29-4a95-a39c-b9ea72ab6873-kube-api-access-wsn5t\") pod \"kserve-controller-manager-d6f4b65b8-xpb5v\" (UID: \"85364ca8-8d29-4a95-a39c-b9ea72ab6873\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xpb5v" Jul 21 00:25:50.245177 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.245152 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85364ca8-8d29-4a95-a39c-b9ea72ab6873-cert\") pod \"kserve-controller-manager-d6f4b65b8-xpb5v\" (UID: \"85364ca8-8d29-4a95-a39c-b9ea72ab6873\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xpb5v" Jul 21 00:25:50.258214 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.258180 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsn5t\" (UniqueName: \"kubernetes.io/projected/85364ca8-8d29-4a95-a39c-b9ea72ab6873-kube-api-access-wsn5t\") pod \"kserve-controller-manager-d6f4b65b8-xpb5v\" (UID: \"85364ca8-8d29-4a95-a39c-b9ea72ab6873\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xpb5v" Jul 21 00:25:50.274566 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.274528 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:50.343620 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.343571 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01b985e5-79f1-4822-b3d5-93bf24f5599e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-s6fs8\" (UID: \"01b985e5-79f1-4822-b3d5-93bf24f5599e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s6fs8" Jul 21 00:25:50.343806 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.343708 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qjgs\" (UniqueName: \"kubernetes.io/projected/01b985e5-79f1-4822-b3d5-93bf24f5599e-kube-api-access-7qjgs\") pod \"llmisvc-controller-manager-777c6bd7d5-s6fs8\" (UID: \"01b985e5-79f1-4822-b3d5-93bf24f5599e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s6fs8" Jul 21 00:25:50.347008 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.346977 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01b985e5-79f1-4822-b3d5-93bf24f5599e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-s6fs8\" (UID: \"01b985e5-79f1-4822-b3d5-93bf24f5599e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s6fs8" Jul 21 00:25:50.362965 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.362933 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qjgs\" (UniqueName: \"kubernetes.io/projected/01b985e5-79f1-4822-b3d5-93bf24f5599e-kube-api-access-7qjgs\") pod \"llmisvc-controller-manager-777c6bd7d5-s6fs8\" (UID: \"01b985e5-79f1-4822-b3d5-93bf24f5599e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s6fs8" Jul 21 00:25:50.425558 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.425522 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:50.455432 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.455388 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xpb5v" Jul 21 00:25:50.463924 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.463893 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s6fs8" Jul 21 00:25:50.511028 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.510982 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:25:50.511221 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.511079 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:25:50.511221 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.511120 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:25:50.617587 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.617558 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xpb5v"] Jul 21 00:25:50.619374 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:25:50.619342 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85364ca8_8d29_4a95_a39c_b9ea72ab6873.slice/crio-3062e9588731365757ea89dc7815d5cfed1c54c30c0d9c89f4f5df2cc70a3d3b WatchSource:0}: Error finding container 3062e9588731365757ea89dc7815d5cfed1c54c30c0d9c89f4f5df2cc70a3d3b: Status 404 returned error can't find the container with id 3062e9588731365757ea89dc7815d5cfed1c54c30c0d9c89f4f5df2cc70a3d3b Jul 21 00:25:50.633006 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:50.632969 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-s6fs8"] Jul 21 00:25:50.636152 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:25:50.636112 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod01b985e5_79f1_4822_b3d5_93bf24f5599e.slice/crio-1b6d72af108ec67664052c59efbc00e91bd72fcc9dbd889755ee0dc16a27d5ab WatchSource:0}: Error finding container 1b6d72af108ec67664052c59efbc00e91bd72fcc9dbd889755ee0dc16a27d5ab: Status 404 returned error can't find the container with id 1b6d72af108ec67664052c59efbc00e91bd72fcc9dbd889755ee0dc16a27d5ab Jul 21 00:25:51.256841 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:51.256807 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s6fs8" event={"ID":"01b985e5-79f1-4822-b3d5-93bf24f5599e","Type":"ContainerStarted","Data":"1b6d72af108ec67664052c59efbc00e91bd72fcc9dbd889755ee0dc16a27d5ab"} Jul 21 00:25:51.258508 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:51.258475 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" event={"ID":"4d01be36-93d9-4180-90cf-ca8f7fde13b9","Type":"ContainerStarted","Data":"99bdf61d1b8350ff967f23f036046ca51c87c6ff18c319112c7cae454660a99d"} Jul 21 00:25:51.259712 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:51.259686 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xpb5v" event={"ID":"85364ca8-8d29-4a95-a39c-b9ea72ab6873","Type":"ContainerStarted","Data":"3062e9588731365757ea89dc7815d5cfed1c54c30c0d9c89f4f5df2cc70a3d3b"} Jul 21 00:25:51.285792 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:51.285693 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" podStartSLOduration=1.726828717 podStartE2EDuration="4.285666956s" totalImagesPullingTime="2.558838239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:25:51.283389389 +0000 UTC m=+545.498769719" watchObservedRunningTime="2026-07-21 00:25:51.285666956 +0000 UTC m=+545.501047281" Jul 21 00:25:51.347702 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:51.347664 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:51.580967 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:51.580926 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:51.717155 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:51.717116 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:51.799059 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:51.798975 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:51.805482 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:51.805449 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:52.264616 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:52.264565 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:52.265644 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:52.265619 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4nzxzx" Jul 21 00:25:54.185534 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.185496 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dknzz"] Jul 21 00:25:54.185939 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.185696 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" podUID="e51af2db-d5f5-4d52-954d-1fcdbb1b3454" containerName="discovery" containerID="cri-o://214b64a29ec9f0aff4290a65f66e4d2342c0d5e434561ee437222d59c55955ef" gracePeriod=30 Jul 21 00:25:54.277611 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.277572 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s6fs8" event={"ID":"01b985e5-79f1-4822-b3d5-93bf24f5599e","Type":"ContainerStarted","Data":"a81ca3be47d296497151abeab6f4c48a50e8a118f7d960cca69381017518725d"} Jul 21 00:25:54.277866 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.277752 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s6fs8" Jul 21 00:25:54.279221 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.279192 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xpb5v" event={"ID":"85364ca8-8d29-4a95-a39c-b9ea72ab6873","Type":"ContainerStarted","Data":"0c7f88ed0799eb740a8ed7bbb0cdd3e7dad35877c8c73c11d41bc4429dda3b5d"} Jul 21 00:25:54.279342 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.279282 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xpb5v" Jul 21 00:25:54.302351 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.302300 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s6fs8" podStartSLOduration=0.90090658 podStartE2EDuration="4.302283693s" totalImagesPullingTime="3.401377113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:25:54.301058991 +0000 UTC m=+548.516439315" watchObservedRunningTime="2026-07-21 00:25:54.302283693 +0000 UTC m=+548.517664018" Jul 21 00:25:54.336437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.336380 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xpb5v" podStartSLOduration=1.59831597 podStartE2EDuration="4.336362365s" totalImagesPullingTime="2.738046395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:25:54.334351774 +0000 UTC m=+548.549732100" watchObservedRunningTime="2026-07-21 00:25:54.336362365 +0000 UTC m=+548.551742690" Jul 21 00:25:54.446025 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.445991 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:54.482346 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.482312 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2wpp\" (UniqueName: \"kubernetes.io/projected/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-kube-api-access-l2wpp\") pod \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " Jul 21 00:25:54.482592 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.482350 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-istio-csr-dns-cert\") pod \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " Jul 21 00:25:54.482592 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.482400 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-local-certs\") pod \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " Jul 21 00:25:54.482592 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.482431 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-cacerts\") pod \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " Jul 21 00:25:54.482592 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.482491 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-istio-token\") pod \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " Jul 21 00:25:54.482592 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.482535 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-istio-kubeconfig\") pod \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " Jul 21 00:25:54.482592 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.482584 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-istio-csr-ca-configmap\") pod \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\" (UID: \"e51af2db-d5f5-4d52-954d-1fcdbb1b3454\") " Jul 21 00:25:54.483231 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.483196 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "e51af2db-d5f5-4d52-954d-1fcdbb1b3454" (UID: "e51af2db-d5f5-4d52-954d-1fcdbb1b3454"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:25:54.485651 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.485593 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-local-certs" (OuterVolumeSpecName: "local-certs") pod "e51af2db-d5f5-4d52-954d-1fcdbb1b3454" (UID: "e51af2db-d5f5-4d52-954d-1fcdbb1b3454"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:25:54.485651 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.485626 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-istio-token" (OuterVolumeSpecName: "istio-token") pod "e51af2db-d5f5-4d52-954d-1fcdbb1b3454" (UID: "e51af2db-d5f5-4d52-954d-1fcdbb1b3454"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:25:54.485651 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.485616 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-kube-api-access-l2wpp" (OuterVolumeSpecName: "kube-api-access-l2wpp") pod "e51af2db-d5f5-4d52-954d-1fcdbb1b3454" (UID: "e51af2db-d5f5-4d52-954d-1fcdbb1b3454"). InnerVolumeSpecName "kube-api-access-l2wpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:25:54.485651 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.485643 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "e51af2db-d5f5-4d52-954d-1fcdbb1b3454" (UID: "e51af2db-d5f5-4d52-954d-1fcdbb1b3454"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:25:54.485838 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.485800 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "e51af2db-d5f5-4d52-954d-1fcdbb1b3454" (UID: "e51af2db-d5f5-4d52-954d-1fcdbb1b3454"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:25:54.485838 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.485817 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-cacerts" (OuterVolumeSpecName: "cacerts") pod "e51af2db-d5f5-4d52-954d-1fcdbb1b3454" (UID: "e51af2db-d5f5-4d52-954d-1fcdbb1b3454"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:25:54.584082 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.584048 2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-istio-csr-ca-configmap\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:25:54.584082 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.584079 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2wpp\" (UniqueName: \"kubernetes.io/projected/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-kube-api-access-l2wpp\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:25:54.584082 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.584089 2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-istio-csr-dns-cert\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:25:54.584321 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.584097 2580 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-local-certs\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:25:54.584321 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.584106 2580 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-cacerts\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:25:54.584321 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.584113 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-istio-token\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:25:54.584321 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:54.584120 2580 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e51af2db-d5f5-4d52-954d-1fcdbb1b3454-istio-kubeconfig\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:25:55.285490 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:55.285456 2580 generic.go:356] "Generic (PLEG): container finished" podID="e51af2db-d5f5-4d52-954d-1fcdbb1b3454" containerID="214b64a29ec9f0aff4290a65f66e4d2342c0d5e434561ee437222d59c55955ef" exitCode=0 Jul 21 00:25:55.285927 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:55.285504 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" Jul 21 00:25:55.285927 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:55.285545 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" event={"ID":"e51af2db-d5f5-4d52-954d-1fcdbb1b3454","Type":"ContainerDied","Data":"214b64a29ec9f0aff4290a65f66e4d2342c0d5e434561ee437222d59c55955ef"} Jul 21 00:25:55.285927 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:55.285595 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-dknzz" event={"ID":"e51af2db-d5f5-4d52-954d-1fcdbb1b3454","Type":"ContainerDied","Data":"57ec430d540076ae0d2c6cdcb241c63e3a05b46cb07c92eac5de6acec7f33a91"} Jul 21 00:25:55.285927 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:55.285620 2580 scope.go:117] "RemoveContainer" containerID="214b64a29ec9f0aff4290a65f66e4d2342c0d5e434561ee437222d59c55955ef" Jul 21 00:25:55.295542 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:55.295522 2580 scope.go:117] "RemoveContainer" containerID="214b64a29ec9f0aff4290a65f66e4d2342c0d5e434561ee437222d59c55955ef" Jul 21 00:25:55.295989 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:25:55.295968 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"214b64a29ec9f0aff4290a65f66e4d2342c0d5e434561ee437222d59c55955ef\": container with ID starting with 214b64a29ec9f0aff4290a65f66e4d2342c0d5e434561ee437222d59c55955ef not found: ID does not exist" containerID="214b64a29ec9f0aff4290a65f66e4d2342c0d5e434561ee437222d59c55955ef" Jul 21 00:25:55.296058 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:55.296001 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"214b64a29ec9f0aff4290a65f66e4d2342c0d5e434561ee437222d59c55955ef"} err="failed to get container status \"214b64a29ec9f0aff4290a65f66e4d2342c0d5e434561ee437222d59c55955ef\": rpc error: code = NotFound desc = could not find container \"214b64a29ec9f0aff4290a65f66e4d2342c0d5e434561ee437222d59c55955ef\": container with ID starting with 214b64a29ec9f0aff4290a65f66e4d2342c0d5e434561ee437222d59c55955ef not found: ID does not exist" Jul 21 00:25:55.314310 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:55.314273 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dknzz"] Jul 21 00:25:55.326254 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:55.326222 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-dknzz"] Jul 21 00:25:56.296306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:56.296272 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e51af2db-d5f5-4d52-954d-1fcdbb1b3454" path="/var/lib/kubelet/pods/e51af2db-d5f5-4d52-954d-1fcdbb1b3454/volumes" Jul 21 00:25:56.342238 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:56.342204 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:25:57.239658 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:57.239632 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-2tln2" Jul 21 00:25:57.287236 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:57.287201 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-h24w6" Jul 21 00:25:58.249946 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:25:58.249910 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:26:01.829824 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:01.829778 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:26:01.881344 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:01.881304 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:26:25.288697 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:25.288666 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-s6fs8" Jul 21 00:26:25.291747 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:25.291715 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xpb5v" Jul 21 00:26:40.170953 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:40.170909 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:26:40.266508 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:40.266474 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:26:40.278215 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:40.278184 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q9pgc"] Jul 21 00:26:40.278507 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:40.278495 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e51af2db-d5f5-4d52-954d-1fcdbb1b3454" containerName="discovery" Jul 21 00:26:40.278549 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:40.278509 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e51af2db-d5f5-4d52-954d-1fcdbb1b3454" containerName="discovery" Jul 21 00:26:40.278586 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:40.278574 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e51af2db-d5f5-4d52-954d-1fcdbb1b3454" containerName="discovery" Jul 21 00:26:40.280878 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:40.280853 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-q9pgc" Jul 21 00:26:40.283273 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:40.283247 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-k266v\"" Jul 21 00:26:40.283420 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:40.283287 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 00:26:40.284169 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:40.284150 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 00:26:40.299133 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:40.299097 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q9pgc"] Jul 21 00:26:40.365340 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:40.365309 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:26:40.474704 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:40.474578 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz59q\" (UniqueName: \"kubernetes.io/projected/77d8fe41-b1e0-40d5-8299-df05a0a419ae-kube-api-access-vz59q\") pod \"kuadrant-operator-catalog-q9pgc\" (UID: \"77d8fe41-b1e0-40d5-8299-df05a0a419ae\") " pod="kuadrant-system/kuadrant-operator-catalog-q9pgc" Jul 21 00:26:40.575819 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:40.575776 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz59q\" (UniqueName: \"kubernetes.io/projected/77d8fe41-b1e0-40d5-8299-df05a0a419ae-kube-api-access-vz59q\") pod \"kuadrant-operator-catalog-q9pgc\" (UID: \"77d8fe41-b1e0-40d5-8299-df05a0a419ae\") " pod="kuadrant-system/kuadrant-operator-catalog-q9pgc" Jul 21 00:26:40.583905 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:40.583879 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz59q\" (UniqueName: \"kubernetes.io/projected/77d8fe41-b1e0-40d5-8299-df05a0a419ae-kube-api-access-vz59q\") pod \"kuadrant-operator-catalog-q9pgc\" (UID: \"77d8fe41-b1e0-40d5-8299-df05a0a419ae\") " pod="kuadrant-system/kuadrant-operator-catalog-q9pgc" Jul 21 00:26:40.593840 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:40.593809 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-q9pgc" Jul 21 00:26:40.650147 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:40.650113 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q9pgc"] Jul 21 00:26:40.731389 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:40.731324 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q9pgc"] Jul 21 00:26:40.733726 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:26:40.733693 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d8fe41_b1e0_40d5_8299_df05a0a419ae.slice/crio-a92fa19461b51c69a51134b5e08751a6b07b1905ad6d9958bf3c1dd17ba4731f WatchSource:0}: Error finding container a92fa19461b51c69a51134b5e08751a6b07b1905ad6d9958bf3c1dd17ba4731f: Status 404 returned error can't find the container with id a92fa19461b51c69a51134b5e08751a6b07b1905ad6d9958bf3c1dd17ba4731f Jul 21 00:26:40.858063 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:40.858029 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zfgtr"] Jul 21 00:26:40.861190 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:40.861169 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zfgtr" Jul 21 00:26:40.869176 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:40.869151 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zfgtr"] Jul 21 00:26:40.978994 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:40.978955 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzzct\" (UniqueName: \"kubernetes.io/projected/477c7451-09be-447d-930b-3c005d6660c7-kube-api-access-bzzct\") pod \"kuadrant-operator-catalog-zfgtr\" (UID: \"477c7451-09be-447d-930b-3c005d6660c7\") " pod="kuadrant-system/kuadrant-operator-catalog-zfgtr" Jul 21 00:26:41.079409 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:41.079375 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzzct\" (UniqueName: \"kubernetes.io/projected/477c7451-09be-447d-930b-3c005d6660c7-kube-api-access-bzzct\") pod \"kuadrant-operator-catalog-zfgtr\" (UID: \"477c7451-09be-447d-930b-3c005d6660c7\") " pod="kuadrant-system/kuadrant-operator-catalog-zfgtr" Jul 21 00:26:41.087695 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:41.087667 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzzct\" (UniqueName: \"kubernetes.io/projected/477c7451-09be-447d-930b-3c005d6660c7-kube-api-access-bzzct\") pod \"kuadrant-operator-catalog-zfgtr\" (UID: \"477c7451-09be-447d-930b-3c005d6660c7\") " pod="kuadrant-system/kuadrant-operator-catalog-zfgtr" Jul 21 00:26:41.172020 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:41.171977 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zfgtr" Jul 21 00:26:41.303991 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:41.303965 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zfgtr"] Jul 21 00:26:41.306319 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:26:41.306290 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod477c7451_09be_447d_930b_3c005d6660c7.slice/crio-623771a91eacaeb3868a00c9f3252a500d2a0578889796b7bc230eca04570d0e WatchSource:0}: Error finding container 623771a91eacaeb3868a00c9f3252a500d2a0578889796b7bc230eca04570d0e: Status 404 returned error can't find the container with id 623771a91eacaeb3868a00c9f3252a500d2a0578889796b7bc230eca04570d0e Jul 21 00:26:41.499993 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:41.499913 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zfgtr" event={"ID":"477c7451-09be-447d-930b-3c005d6660c7","Type":"ContainerStarted","Data":"623771a91eacaeb3868a00c9f3252a500d2a0578889796b7bc230eca04570d0e"} Jul 21 00:26:41.501015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:41.500991 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-q9pgc" event={"ID":"77d8fe41-b1e0-40d5-8299-df05a0a419ae","Type":"ContainerStarted","Data":"a92fa19461b51c69a51134b5e08751a6b07b1905ad6d9958bf3c1dd17ba4731f"} Jul 21 00:26:43.518513 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:43.518465 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zfgtr" event={"ID":"477c7451-09be-447d-930b-3c005d6660c7","Type":"ContainerStarted","Data":"1439c9d80411302a7aaa65c343d77a3fe9806fcdcc8cc7aa59b36a8669fa0500"} Jul 21 00:26:43.520197 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:43.520169 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-q9pgc" event={"ID":"77d8fe41-b1e0-40d5-8299-df05a0a419ae","Type":"ContainerStarted","Data":"4146993d2cefda0011912ba2eb547e4ad7e861fa4058efa4678523b03a3472c3"} Jul 21 00:26:43.520320 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:43.520196 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-q9pgc" podUID="77d8fe41-b1e0-40d5-8299-df05a0a419ae" containerName="registry-server" containerID="cri-o://4146993d2cefda0011912ba2eb547e4ad7e861fa4058efa4678523b03a3472c3" gracePeriod=2 Jul 21 00:26:43.535902 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:43.535841 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-zfgtr" podStartSLOduration=1.473807475 podStartE2EDuration="3.535822447s" totalImagesPullingTime="2.062014972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:26:43.53368078 +0000 UTC m=+597.749061107" watchObservedRunningTime="2026-07-21 00:26:43.535822447 +0000 UTC m=+597.751202771" Jul 21 00:26:43.547843 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:43.547626 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-q9pgc" podStartSLOduration=0.91534267 podStartE2EDuration="3.547591402s" totalImagesPullingTime="2.632248732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:26:43.547165299 +0000 UTC m=+597.762545625" watchObservedRunningTime="2026-07-21 00:26:43.547591402 +0000 UTC m=+597.762971727" Jul 21 00:26:43.763304 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:43.763280 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-q9pgc_77d8fe41-b1e0-40d5-8299-df05a0a419ae/registry-server/0.log" Jul 21 00:26:43.765618 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:43.765581 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-q9pgc" Jul 21 00:26:43.799798 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:43.799757 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vz59q\" (UniqueName: \"kubernetes.io/projected/77d8fe41-b1e0-40d5-8299-df05a0a419ae-kube-api-access-vz59q\") pod \"77d8fe41-b1e0-40d5-8299-df05a0a419ae\" (UID: \"77d8fe41-b1e0-40d5-8299-df05a0a419ae\") " Jul 21 00:26:43.802135 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:43.802097 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d8fe41-b1e0-40d5-8299-df05a0a419ae-kube-api-access-vz59q" (OuterVolumeSpecName: "kube-api-access-vz59q") pod "77d8fe41-b1e0-40d5-8299-df05a0a419ae" (UID: "77d8fe41-b1e0-40d5-8299-df05a0a419ae"). InnerVolumeSpecName "kube-api-access-vz59q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:26:43.901270 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:43.901174 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz59q\" (UniqueName: \"kubernetes.io/projected/77d8fe41-b1e0-40d5-8299-df05a0a419ae-kube-api-access-vz59q\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:26:44.527347 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:44.527316 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-q9pgc_77d8fe41-b1e0-40d5-8299-df05a0a419ae/registry-server/0.log" Jul 21 00:26:44.527846 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:44.527361 2580 generic.go:356] "Generic (PLEG): container finished" podID="77d8fe41-b1e0-40d5-8299-df05a0a419ae" containerID="4146993d2cefda0011912ba2eb547e4ad7e861fa4058efa4678523b03a3472c3" exitCode=1 Jul 21 00:26:44.527846 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:44.527413 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-q9pgc" Jul 21 00:26:44.527846 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:44.527447 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-q9pgc" event={"ID":"77d8fe41-b1e0-40d5-8299-df05a0a419ae","Type":"ContainerDied","Data":"4146993d2cefda0011912ba2eb547e4ad7e861fa4058efa4678523b03a3472c3"} Jul 21 00:26:44.527846 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:44.527495 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-q9pgc" event={"ID":"77d8fe41-b1e0-40d5-8299-df05a0a419ae","Type":"ContainerDied","Data":"a92fa19461b51c69a51134b5e08751a6b07b1905ad6d9958bf3c1dd17ba4731f"} Jul 21 00:26:44.527846 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:44.527522 2580 scope.go:117] "RemoveContainer" containerID="4146993d2cefda0011912ba2eb547e4ad7e861fa4058efa4678523b03a3472c3" Jul 21 00:26:44.536855 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:44.536837 2580 scope.go:117] "RemoveContainer" containerID="4146993d2cefda0011912ba2eb547e4ad7e861fa4058efa4678523b03a3472c3" Jul 21 00:26:44.537197 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:26:44.537173 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4146993d2cefda0011912ba2eb547e4ad7e861fa4058efa4678523b03a3472c3\": container with ID starting with 4146993d2cefda0011912ba2eb547e4ad7e861fa4058efa4678523b03a3472c3 not found: ID does not exist" containerID="4146993d2cefda0011912ba2eb547e4ad7e861fa4058efa4678523b03a3472c3" Jul 21 00:26:44.537267 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:44.537209 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4146993d2cefda0011912ba2eb547e4ad7e861fa4058efa4678523b03a3472c3"} err="failed to get container status \"4146993d2cefda0011912ba2eb547e4ad7e861fa4058efa4678523b03a3472c3\": rpc error: code = NotFound desc = could not find container \"4146993d2cefda0011912ba2eb547e4ad7e861fa4058efa4678523b03a3472c3\": container with ID starting with 4146993d2cefda0011912ba2eb547e4ad7e861fa4058efa4678523b03a3472c3 not found: ID does not exist" Jul 21 00:26:44.544486 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:44.544451 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q9pgc"] Jul 21 00:26:44.546445 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:44.546416 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q9pgc"] Jul 21 00:26:46.297305 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:46.297194 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77d8fe41-b1e0-40d5-8299-df05a0a419ae" path="/var/lib/kubelet/pods/77d8fe41-b1e0-40d5-8299-df05a0a419ae/volumes" Jul 21 00:26:51.172930 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:51.172878 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-zfgtr" Jul 21 00:26:51.173341 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:51.172956 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-zfgtr" Jul 21 00:26:51.196216 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:51.196190 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-zfgtr" Jul 21 00:26:51.581322 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:51.581294 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-zfgtr" Jul 21 00:26:55.457078 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:55.457026 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms"] Jul 21 00:26:55.457502 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:55.457361 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77d8fe41-b1e0-40d5-8299-df05a0a419ae" containerName="registry-server" Jul 21 00:26:55.457502 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:55.457371 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="77d8fe41-b1e0-40d5-8299-df05a0a419ae" containerName="registry-server" Jul 21 00:26:55.457502 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:55.457423 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="77d8fe41-b1e0-40d5-8299-df05a0a419ae" containerName="registry-server" Jul 21 00:26:55.460581 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:55.460563 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms" Jul 21 00:26:55.463155 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:55.463134 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xfbv5\"" Jul 21 00:26:55.469084 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:55.469058 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms"] Jul 21 00:26:55.510120 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:55.510074 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhqdd\" (UniqueName: \"kubernetes.io/projected/b6a041bf-6dae-4b00-9fb7-30b07b16bdbd-kube-api-access-dhqdd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms\" (UID: \"b6a041bf-6dae-4b00-9fb7-30b07b16bdbd\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms" Jul 21 00:26:55.510120 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:55.510115 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6a041bf-6dae-4b00-9fb7-30b07b16bdbd-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms\" (UID: \"b6a041bf-6dae-4b00-9fb7-30b07b16bdbd\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms" Jul 21 00:26:55.510357 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:55.510180 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6a041bf-6dae-4b00-9fb7-30b07b16bdbd-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms\" (UID: \"b6a041bf-6dae-4b00-9fb7-30b07b16bdbd\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms" Jul 21 00:26:55.610875 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:55.610837 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6a041bf-6dae-4b00-9fb7-30b07b16bdbd-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms\" (UID: \"b6a041bf-6dae-4b00-9fb7-30b07b16bdbd\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms" Jul 21 00:26:55.611087 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:55.610931 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhqdd\" (UniqueName: \"kubernetes.io/projected/b6a041bf-6dae-4b00-9fb7-30b07b16bdbd-kube-api-access-dhqdd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms\" (UID: \"b6a041bf-6dae-4b00-9fb7-30b07b16bdbd\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms" Jul 21 00:26:55.611087 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:55.610959 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6a041bf-6dae-4b00-9fb7-30b07b16bdbd-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms\" (UID: \"b6a041bf-6dae-4b00-9fb7-30b07b16bdbd\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms" Jul 21 00:26:55.611249 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:55.611226 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6a041bf-6dae-4b00-9fb7-30b07b16bdbd-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms\" (UID: \"b6a041bf-6dae-4b00-9fb7-30b07b16bdbd\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms" Jul 21 00:26:55.611330 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:55.611309 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6a041bf-6dae-4b00-9fb7-30b07b16bdbd-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms\" (UID: \"b6a041bf-6dae-4b00-9fb7-30b07b16bdbd\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms" Jul 21 00:26:55.618766 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:55.618732 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhqdd\" (UniqueName: \"kubernetes.io/projected/b6a041bf-6dae-4b00-9fb7-30b07b16bdbd-kube-api-access-dhqdd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms\" (UID: \"b6a041bf-6dae-4b00-9fb7-30b07b16bdbd\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms" Jul 21 00:26:55.771164 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:55.771059 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms" Jul 21 00:26:55.913318 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:55.913285 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms"] Jul 21 00:26:55.916025 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:26:55.915991 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a041bf_6dae_4b00_9fb7_30b07b16bdbd.slice/crio-1df133051264cc26ffaa8a5712e3806a56ca66ac4e2255a526d13b62f205eaeb WatchSource:0}: Error finding container 1df133051264cc26ffaa8a5712e3806a56ca66ac4e2255a526d13b62f205eaeb: Status 404 returned error can't find the container with id 1df133051264cc26ffaa8a5712e3806a56ca66ac4e2255a526d13b62f205eaeb Jul 21 00:26:56.063064 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.063029 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl"] Jul 21 00:26:56.066412 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.066392 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl" Jul 21 00:26:56.074408 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.074377 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl"] Jul 21 00:26:56.116318 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.116274 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd296917-fa8f-471e-b4fc-9795c3be221d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl\" (UID: \"dd296917-fa8f-471e-b4fc-9795c3be221d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl" Jul 21 00:26:56.116318 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.116322 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd296917-fa8f-471e-b4fc-9795c3be221d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl\" (UID: \"dd296917-fa8f-471e-b4fc-9795c3be221d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl" Jul 21 00:26:56.116546 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.116394 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps4zq\" (UniqueName: \"kubernetes.io/projected/dd296917-fa8f-471e-b4fc-9795c3be221d-kube-api-access-ps4zq\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl\" (UID: \"dd296917-fa8f-471e-b4fc-9795c3be221d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl" Jul 21 00:26:56.217289 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.217248 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps4zq\" (UniqueName: \"kubernetes.io/projected/dd296917-fa8f-471e-b4fc-9795c3be221d-kube-api-access-ps4zq\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl\" (UID: \"dd296917-fa8f-471e-b4fc-9795c3be221d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl" Jul 21 00:26:56.217478 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.217297 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd296917-fa8f-471e-b4fc-9795c3be221d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl\" (UID: \"dd296917-fa8f-471e-b4fc-9795c3be221d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl" Jul 21 00:26:56.217478 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.217336 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd296917-fa8f-471e-b4fc-9795c3be221d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl\" (UID: \"dd296917-fa8f-471e-b4fc-9795c3be221d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl" Jul 21 00:26:56.217738 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.217719 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd296917-fa8f-471e-b4fc-9795c3be221d-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl\" (UID: \"dd296917-fa8f-471e-b4fc-9795c3be221d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl" Jul 21 00:26:56.217789 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.217761 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd296917-fa8f-471e-b4fc-9795c3be221d-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl\" (UID: \"dd296917-fa8f-471e-b4fc-9795c3be221d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl" Jul 21 00:26:56.225412 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.225382 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps4zq\" (UniqueName: \"kubernetes.io/projected/dd296917-fa8f-471e-b4fc-9795c3be221d-kube-api-access-ps4zq\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl\" (UID: \"dd296917-fa8f-471e-b4fc-9795c3be221d\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl" Jul 21 00:26:56.402770 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.402731 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl" Jul 21 00:26:56.543727 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.543686 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl"] Jul 21 00:26:56.545998 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:26:56.545953 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd296917_fa8f_471e_b4fc_9795c3be221d.slice/crio-357636c76216466f614a0a36dff3bd8187a550ae9287d2d2e31e0a55be254dd1 WatchSource:0}: Error finding container 357636c76216466f614a0a36dff3bd8187a550ae9287d2d2e31e0a55be254dd1: Status 404 returned error can't find the container with id 357636c76216466f614a0a36dff3bd8187a550ae9287d2d2e31e0a55be254dd1 Jul 21 00:26:56.582964 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.582930 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl" event={"ID":"dd296917-fa8f-471e-b4fc-9795c3be221d","Type":"ContainerStarted","Data":"357636c76216466f614a0a36dff3bd8187a550ae9287d2d2e31e0a55be254dd1"} Jul 21 00:26:56.584671 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.584629 2580 generic.go:356] "Generic (PLEG): container finished" podID="b6a041bf-6dae-4b00-9fb7-30b07b16bdbd" containerID="08d85ec7095aee4c81581d92df2c960402cf2bd4869757f56d348a36c4d44746" exitCode=0 Jul 21 00:26:56.584830 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.584684 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms" event={"ID":"b6a041bf-6dae-4b00-9fb7-30b07b16bdbd","Type":"ContainerDied","Data":"08d85ec7095aee4c81581d92df2c960402cf2bd4869757f56d348a36c4d44746"} Jul 21 00:26:56.584830 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.584726 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms" event={"ID":"b6a041bf-6dae-4b00-9fb7-30b07b16bdbd","Type":"ContainerStarted","Data":"1df133051264cc26ffaa8a5712e3806a56ca66ac4e2255a526d13b62f205eaeb"} Jul 21 00:26:56.675979 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.675940 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5944955d98-bhzfc"] Jul 21 00:26:56.679191 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.679165 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:26:56.680357 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.680332 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn"] Jul 21 00:26:56.683750 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.683727 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn" Jul 21 00:26:56.695490 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.695458 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5944955d98-bhzfc"] Jul 21 00:26:56.709146 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.709112 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn"] Jul 21 00:26:56.722364 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.722331 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9351e4d7-fd14-498b-b85f-8a714dcf8df6-service-ca\") pod \"console-5944955d98-bhzfc\" (UID: \"9351e4d7-fd14-498b-b85f-8a714dcf8df6\") " pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:26:56.722364 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.722367 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwzc2\" (UniqueName: \"kubernetes.io/projected/9351e4d7-fd14-498b-b85f-8a714dcf8df6-kube-api-access-cwzc2\") pod \"console-5944955d98-bhzfc\" (UID: \"9351e4d7-fd14-498b-b85f-8a714dcf8df6\") " pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:26:56.722631 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.722410 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9351e4d7-fd14-498b-b85f-8a714dcf8df6-trusted-ca-bundle\") pod \"console-5944955d98-bhzfc\" (UID: \"9351e4d7-fd14-498b-b85f-8a714dcf8df6\") " pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:26:56.722631 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.722445 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d48e193a-a228-4bfa-8bb0-b756f456bf10-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn\" (UID: \"d48e193a-a228-4bfa-8bb0-b756f456bf10\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn" Jul 21 00:26:56.722631 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.722469 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9351e4d7-fd14-498b-b85f-8a714dcf8df6-console-serving-cert\") pod \"console-5944955d98-bhzfc\" (UID: \"9351e4d7-fd14-498b-b85f-8a714dcf8df6\") " pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:26:56.722631 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.722496 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9351e4d7-fd14-498b-b85f-8a714dcf8df6-console-oauth-config\") pod \"console-5944955d98-bhzfc\" (UID: \"9351e4d7-fd14-498b-b85f-8a714dcf8df6\") " pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:26:56.722631 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.722513 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9351e4d7-fd14-498b-b85f-8a714dcf8df6-console-config\") pod \"console-5944955d98-bhzfc\" (UID: \"9351e4d7-fd14-498b-b85f-8a714dcf8df6\") " pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:26:56.722631 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.722536 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d48e193a-a228-4bfa-8bb0-b756f456bf10-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn\" (UID: \"d48e193a-a228-4bfa-8bb0-b756f456bf10\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn" Jul 21 00:26:56.722880 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.722677 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9351e4d7-fd14-498b-b85f-8a714dcf8df6-oauth-serving-cert\") pod \"console-5944955d98-bhzfc\" (UID: \"9351e4d7-fd14-498b-b85f-8a714dcf8df6\") " pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:26:56.722880 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.722726 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q696k\" (UniqueName: \"kubernetes.io/projected/d48e193a-a228-4bfa-8bb0-b756f456bf10-kube-api-access-q696k\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn\" (UID: \"d48e193a-a228-4bfa-8bb0-b756f456bf10\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn" Jul 21 00:26:56.746047 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.746019 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:26:56.813264 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.813207 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:26:56.823337 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.823295 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9351e4d7-fd14-498b-b85f-8a714dcf8df6-trusted-ca-bundle\") pod \"console-5944955d98-bhzfc\" (UID: \"9351e4d7-fd14-498b-b85f-8a714dcf8df6\") " pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:26:56.823527 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.823356 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d48e193a-a228-4bfa-8bb0-b756f456bf10-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn\" (UID: \"d48e193a-a228-4bfa-8bb0-b756f456bf10\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn" Jul 21 00:26:56.823527 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.823389 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9351e4d7-fd14-498b-b85f-8a714dcf8df6-console-serving-cert\") pod \"console-5944955d98-bhzfc\" (UID: \"9351e4d7-fd14-498b-b85f-8a714dcf8df6\") " pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:26:56.823527 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.823425 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9351e4d7-fd14-498b-b85f-8a714dcf8df6-console-oauth-config\") pod \"console-5944955d98-bhzfc\" (UID: \"9351e4d7-fd14-498b-b85f-8a714dcf8df6\") " pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:26:56.823527 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.823445 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9351e4d7-fd14-498b-b85f-8a714dcf8df6-console-config\") pod \"console-5944955d98-bhzfc\" (UID: \"9351e4d7-fd14-498b-b85f-8a714dcf8df6\") " pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:26:56.823527 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.823473 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d48e193a-a228-4bfa-8bb0-b756f456bf10-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn\" (UID: \"d48e193a-a228-4bfa-8bb0-b756f456bf10\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn" Jul 21 00:26:56.823527 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.823513 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9351e4d7-fd14-498b-b85f-8a714dcf8df6-oauth-serving-cert\") pod \"console-5944955d98-bhzfc\" (UID: \"9351e4d7-fd14-498b-b85f-8a714dcf8df6\") " pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:26:56.824068 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.823539 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q696k\" (UniqueName: \"kubernetes.io/projected/d48e193a-a228-4bfa-8bb0-b756f456bf10-kube-api-access-q696k\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn\" (UID: \"d48e193a-a228-4bfa-8bb0-b756f456bf10\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn" Jul 21 00:26:56.824068 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.823587 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9351e4d7-fd14-498b-b85f-8a714dcf8df6-service-ca\") pod \"console-5944955d98-bhzfc\" (UID: \"9351e4d7-fd14-498b-b85f-8a714dcf8df6\") " pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:26:56.824068 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.823634 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwzc2\" (UniqueName: \"kubernetes.io/projected/9351e4d7-fd14-498b-b85f-8a714dcf8df6-kube-api-access-cwzc2\") pod \"console-5944955d98-bhzfc\" (UID: \"9351e4d7-fd14-498b-b85f-8a714dcf8df6\") " pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:26:56.824068 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.823872 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d48e193a-a228-4bfa-8bb0-b756f456bf10-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn\" (UID: \"d48e193a-a228-4bfa-8bb0-b756f456bf10\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn" Jul 21 00:26:56.824068 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.823950 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d48e193a-a228-4bfa-8bb0-b756f456bf10-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn\" (UID: \"d48e193a-a228-4bfa-8bb0-b756f456bf10\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn" Jul 21 00:26:56.824814 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.824362 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9351e4d7-fd14-498b-b85f-8a714dcf8df6-oauth-serving-cert\") pod \"console-5944955d98-bhzfc\" (UID: \"9351e4d7-fd14-498b-b85f-8a714dcf8df6\") " pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:26:56.824814 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.824368 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9351e4d7-fd14-498b-b85f-8a714dcf8df6-trusted-ca-bundle\") pod \"console-5944955d98-bhzfc\" (UID: \"9351e4d7-fd14-498b-b85f-8a714dcf8df6\") " pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:26:56.824814 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.824415 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9351e4d7-fd14-498b-b85f-8a714dcf8df6-console-config\") pod \"console-5944955d98-bhzfc\" (UID: \"9351e4d7-fd14-498b-b85f-8a714dcf8df6\") " pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:26:56.824814 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.824481 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9351e4d7-fd14-498b-b85f-8a714dcf8df6-service-ca\") pod \"console-5944955d98-bhzfc\" (UID: \"9351e4d7-fd14-498b-b85f-8a714dcf8df6\") " pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:26:56.826137 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.826118 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9351e4d7-fd14-498b-b85f-8a714dcf8df6-console-serving-cert\") pod \"console-5944955d98-bhzfc\" (UID: \"9351e4d7-fd14-498b-b85f-8a714dcf8df6\") " pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:26:56.826355 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.826340 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9351e4d7-fd14-498b-b85f-8a714dcf8df6-console-oauth-config\") pod \"console-5944955d98-bhzfc\" (UID: \"9351e4d7-fd14-498b-b85f-8a714dcf8df6\") " pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:26:56.833003 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.832981 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwzc2\" (UniqueName: \"kubernetes.io/projected/9351e4d7-fd14-498b-b85f-8a714dcf8df6-kube-api-access-cwzc2\") pod \"console-5944955d98-bhzfc\" (UID: \"9351e4d7-fd14-498b-b85f-8a714dcf8df6\") " pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:26:56.833120 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:56.833045 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q696k\" (UniqueName: \"kubernetes.io/projected/d48e193a-a228-4bfa-8bb0-b756f456bf10-kube-api-access-q696k\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn\" (UID: \"d48e193a-a228-4bfa-8bb0-b756f456bf10\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn" Jul 21 00:26:57.048380 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.048292 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:26:57.052679 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.052655 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn" Jul 21 00:26:57.062269 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.062240 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7"] Jul 21 00:26:57.067589 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.067566 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7" Jul 21 00:26:57.074056 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.074019 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7"] Jul 21 00:26:57.126152 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.126113 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-962kc\" (UniqueName: \"kubernetes.io/projected/1f66e031-60a1-4f6d-942b-782770b2ea3d-kube-api-access-962kc\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7\" (UID: \"1f66e031-60a1-4f6d-942b-782770b2ea3d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7" Jul 21 00:26:57.126341 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.126175 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f66e031-60a1-4f6d-942b-782770b2ea3d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7\" (UID: \"1f66e031-60a1-4f6d-942b-782770b2ea3d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7" Jul 21 00:26:57.126341 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.126251 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f66e031-60a1-4f6d-942b-782770b2ea3d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7\" (UID: \"1f66e031-60a1-4f6d-942b-782770b2ea3d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7" Jul 21 00:26:57.199184 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.199159 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn"] Jul 21 00:26:57.200750 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:26:57.200726 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48e193a_a228_4bfa_8bb0_b756f456bf10.slice/crio-33521aa241a525b99364ca5140c7b61237350d62758c35a02a05df75f76bfd8c WatchSource:0}: Error finding container 33521aa241a525b99364ca5140c7b61237350d62758c35a02a05df75f76bfd8c: Status 404 returned error can't find the container with id 33521aa241a525b99364ca5140c7b61237350d62758c35a02a05df75f76bfd8c Jul 21 00:26:57.218048 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.218027 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5944955d98-bhzfc"] Jul 21 00:26:57.220694 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:26:57.220662 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9351e4d7_fd14_498b_b85f_8a714dcf8df6.slice/crio-40ec49a44cb45617351363e066c95275c0ffa4c2ff8c49bb5726cc158c3c940f WatchSource:0}: Error finding container 40ec49a44cb45617351363e066c95275c0ffa4c2ff8c49bb5726cc158c3c940f: Status 404 returned error can't find the container with id 40ec49a44cb45617351363e066c95275c0ffa4c2ff8c49bb5726cc158c3c940f Jul 21 00:26:57.226967 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.226940 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f66e031-60a1-4f6d-942b-782770b2ea3d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7\" (UID: \"1f66e031-60a1-4f6d-942b-782770b2ea3d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7" Jul 21 00:26:57.227053 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.226995 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-962kc\" (UniqueName: \"kubernetes.io/projected/1f66e031-60a1-4f6d-942b-782770b2ea3d-kube-api-access-962kc\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7\" (UID: \"1f66e031-60a1-4f6d-942b-782770b2ea3d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7" Jul 21 00:26:57.227053 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.227023 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f66e031-60a1-4f6d-942b-782770b2ea3d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7\" (UID: \"1f66e031-60a1-4f6d-942b-782770b2ea3d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7" Jul 21 00:26:57.227382 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.227359 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f66e031-60a1-4f6d-942b-782770b2ea3d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7\" (UID: \"1f66e031-60a1-4f6d-942b-782770b2ea3d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7" Jul 21 00:26:57.227422 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.227385 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f66e031-60a1-4f6d-942b-782770b2ea3d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7\" (UID: \"1f66e031-60a1-4f6d-942b-782770b2ea3d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7" Jul 21 00:26:57.235656 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.235621 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-962kc\" (UniqueName: \"kubernetes.io/projected/1f66e031-60a1-4f6d-942b-782770b2ea3d-kube-api-access-962kc\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7\" (UID: \"1f66e031-60a1-4f6d-942b-782770b2ea3d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7" Jul 21 00:26:57.380688 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.380652 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7" Jul 21 00:26:57.519401 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.519370 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7"] Jul 21 00:26:57.521044 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:26:57.521018 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f66e031_60a1_4f6d_942b_782770b2ea3d.slice/crio-89a96dec8f3b37c1a0c5c7654aa4352c2673601a16d7976328ffe006dc86ca88 WatchSource:0}: Error finding container 89a96dec8f3b37c1a0c5c7654aa4352c2673601a16d7976328ffe006dc86ca88: Status 404 returned error can't find the container with id 89a96dec8f3b37c1a0c5c7654aa4352c2673601a16d7976328ffe006dc86ca88 Jul 21 00:26:57.592038 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.592002 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5944955d98-bhzfc" event={"ID":"9351e4d7-fd14-498b-b85f-8a714dcf8df6","Type":"ContainerStarted","Data":"2cb7d11f60ec22df0030d7d6fbb9d0918668abe4f0e977d02e2fd075f6e22937"} Jul 21 00:26:57.592526 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.592056 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5944955d98-bhzfc" event={"ID":"9351e4d7-fd14-498b-b85f-8a714dcf8df6","Type":"ContainerStarted","Data":"40ec49a44cb45617351363e066c95275c0ffa4c2ff8c49bb5726cc158c3c940f"} Jul 21 00:26:57.593725 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.593700 2580 generic.go:356] "Generic (PLEG): container finished" podID="dd296917-fa8f-471e-b4fc-9795c3be221d" containerID="0bf3738a479f9cc114a6d980849bbdb6c7e4602849f19e871a309cc5c01b4ff9" exitCode=0 Jul 21 00:26:57.593842 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.593782 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl" event={"ID":"dd296917-fa8f-471e-b4fc-9795c3be221d","Type":"ContainerDied","Data":"0bf3738a479f9cc114a6d980849bbdb6c7e4602849f19e871a309cc5c01b4ff9"} Jul 21 00:26:57.595541 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.595508 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7" event={"ID":"1f66e031-60a1-4f6d-942b-782770b2ea3d","Type":"ContainerStarted","Data":"0c5bc55a82996f4715a17b2daf1db063c321bbf78ac006bb923df6f5fdc73d7e"} Jul 21 00:26:57.595669 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.595558 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7" event={"ID":"1f66e031-60a1-4f6d-942b-782770b2ea3d","Type":"ContainerStarted","Data":"89a96dec8f3b37c1a0c5c7654aa4352c2673601a16d7976328ffe006dc86ca88"} Jul 21 00:26:57.597177 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.597153 2580 generic.go:356] "Generic (PLEG): container finished" podID="d48e193a-a228-4bfa-8bb0-b756f456bf10" containerID="f0ba1acfb8d7bfee28ee5a4f598f5a2aac59bd020f9f67be18277f24d7598473" exitCode=0 Jul 21 00:26:57.597249 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.597215 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn" event={"ID":"d48e193a-a228-4bfa-8bb0-b756f456bf10","Type":"ContainerDied","Data":"f0ba1acfb8d7bfee28ee5a4f598f5a2aac59bd020f9f67be18277f24d7598473"} Jul 21 00:26:57.597283 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.597251 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn" event={"ID":"d48e193a-a228-4bfa-8bb0-b756f456bf10","Type":"ContainerStarted","Data":"33521aa241a525b99364ca5140c7b61237350d62758c35a02a05df75f76bfd8c"} Jul 21 00:26:57.615069 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:57.614995 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5944955d98-bhzfc" podStartSLOduration=1.6149787039999999 podStartE2EDuration="1.614978704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:26:57.613752174 +0000 UTC m=+611.829132501" watchObservedRunningTime="2026-07-21 00:26:57.614978704 +0000 UTC m=+611.830359030" Jul 21 00:26:58.604418 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:58.604373 2580 generic.go:356] "Generic (PLEG): container finished" podID="d48e193a-a228-4bfa-8bb0-b756f456bf10" containerID="67476fa3c8b865c32d0c2a98053c6fe26a262b983c4fdce413d42d9b21448e62" exitCode=0 Jul 21 00:26:58.604835 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:58.604503 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn" event={"ID":"d48e193a-a228-4bfa-8bb0-b756f456bf10","Type":"ContainerDied","Data":"67476fa3c8b865c32d0c2a98053c6fe26a262b983c4fdce413d42d9b21448e62"} Jul 21 00:26:58.606541 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:58.606518 2580 generic.go:356] "Generic (PLEG): container finished" podID="b6a041bf-6dae-4b00-9fb7-30b07b16bdbd" containerID="ab7ce0ed13c25bd0b39fee7ab0119e292e100fa6cb7c0f4feafc7d00724832a0" exitCode=0 Jul 21 00:26:58.606680 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:58.606589 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms" event={"ID":"b6a041bf-6dae-4b00-9fb7-30b07b16bdbd","Type":"ContainerDied","Data":"ab7ce0ed13c25bd0b39fee7ab0119e292e100fa6cb7c0f4feafc7d00724832a0"} Jul 21 00:26:58.608052 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:58.608030 2580 generic.go:356] "Generic (PLEG): container finished" podID="1f66e031-60a1-4f6d-942b-782770b2ea3d" containerID="0c5bc55a82996f4715a17b2daf1db063c321bbf78ac006bb923df6f5fdc73d7e" exitCode=0 Jul 21 00:26:58.608159 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:58.608088 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7" event={"ID":"1f66e031-60a1-4f6d-942b-782770b2ea3d","Type":"ContainerDied","Data":"0c5bc55a82996f4715a17b2daf1db063c321bbf78ac006bb923df6f5fdc73d7e"} Jul 21 00:26:59.618665 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:59.618627 2580 generic.go:356] "Generic (PLEG): container finished" podID="dd296917-fa8f-471e-b4fc-9795c3be221d" containerID="c76ee30f022a723fbea194ab4ef74331c8dab83cd85ae797cfb1a1a306245dff" exitCode=0 Jul 21 00:26:59.619144 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:59.618702 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl" event={"ID":"dd296917-fa8f-471e-b4fc-9795c3be221d","Type":"ContainerDied","Data":"c76ee30f022a723fbea194ab4ef74331c8dab83cd85ae797cfb1a1a306245dff"} Jul 21 00:26:59.620967 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:59.620940 2580 generic.go:356] "Generic (PLEG): container finished" podID="b6a041bf-6dae-4b00-9fb7-30b07b16bdbd" containerID="16c42a4284bbc6d7dc9369001769e622106b669deaa6fe8b7f74f808a676f22d" exitCode=0 Jul 21 00:26:59.621092 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:59.621018 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms" event={"ID":"b6a041bf-6dae-4b00-9fb7-30b07b16bdbd","Type":"ContainerDied","Data":"16c42a4284bbc6d7dc9369001769e622106b669deaa6fe8b7f74f808a676f22d"} Jul 21 00:26:59.623264 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:59.623236 2580 generic.go:356] "Generic (PLEG): container finished" podID="d48e193a-a228-4bfa-8bb0-b756f456bf10" containerID="6116e749f7d198d6c476492c41d799ea29b572e310b261fddca745a94ba69c45" exitCode=0 Jul 21 00:26:59.623388 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:26:59.623309 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn" event={"ID":"d48e193a-a228-4bfa-8bb0-b756f456bf10","Type":"ContainerDied","Data":"6116e749f7d198d6c476492c41d799ea29b572e310b261fddca745a94ba69c45"} Jul 21 00:27:00.631066 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:00.631034 2580 generic.go:356] "Generic (PLEG): container finished" podID="dd296917-fa8f-471e-b4fc-9795c3be221d" containerID="449b67c7ce4215e72a7c88099a4ff574b6e07a36828faf268bdce1c296ddf12b" exitCode=0 Jul 21 00:27:00.631546 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:00.631130 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl" event={"ID":"dd296917-fa8f-471e-b4fc-9795c3be221d","Type":"ContainerDied","Data":"449b67c7ce4215e72a7c88099a4ff574b6e07a36828faf268bdce1c296ddf12b"} Jul 21 00:27:00.632912 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:00.632890 2580 generic.go:356] "Generic (PLEG): container finished" podID="1f66e031-60a1-4f6d-942b-782770b2ea3d" containerID="0fdfed032529ca1d2f3782c526feaaaf9c0ce270af08bf3f1a0c9fa45339fa04" exitCode=0 Jul 21 00:27:00.633038 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:00.632969 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7" event={"ID":"1f66e031-60a1-4f6d-942b-782770b2ea3d","Type":"ContainerDied","Data":"0fdfed032529ca1d2f3782c526feaaaf9c0ce270af08bf3f1a0c9fa45339fa04"} Jul 21 00:27:00.797183 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:00.797157 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms" Jul 21 00:27:00.867336 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:00.867298 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6a041bf-6dae-4b00-9fb7-30b07b16bdbd-bundle\") pod \"b6a041bf-6dae-4b00-9fb7-30b07b16bdbd\" (UID: \"b6a041bf-6dae-4b00-9fb7-30b07b16bdbd\") " Jul 21 00:27:00.867527 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:00.867401 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhqdd\" (UniqueName: \"kubernetes.io/projected/b6a041bf-6dae-4b00-9fb7-30b07b16bdbd-kube-api-access-dhqdd\") pod \"b6a041bf-6dae-4b00-9fb7-30b07b16bdbd\" (UID: \"b6a041bf-6dae-4b00-9fb7-30b07b16bdbd\") " Jul 21 00:27:00.867527 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:00.867465 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6a041bf-6dae-4b00-9fb7-30b07b16bdbd-util\") pod \"b6a041bf-6dae-4b00-9fb7-30b07b16bdbd\" (UID: \"b6a041bf-6dae-4b00-9fb7-30b07b16bdbd\") " Jul 21 00:27:00.867955 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:00.867913 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a041bf-6dae-4b00-9fb7-30b07b16bdbd-bundle" (OuterVolumeSpecName: "bundle") pod "b6a041bf-6dae-4b00-9fb7-30b07b16bdbd" (UID: "b6a041bf-6dae-4b00-9fb7-30b07b16bdbd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:27:00.869662 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:00.869634 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a041bf-6dae-4b00-9fb7-30b07b16bdbd-kube-api-access-dhqdd" (OuterVolumeSpecName: "kube-api-access-dhqdd") pod "b6a041bf-6dae-4b00-9fb7-30b07b16bdbd" (UID: "b6a041bf-6dae-4b00-9fb7-30b07b16bdbd"). InnerVolumeSpecName "kube-api-access-dhqdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:27:00.873209 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:00.873182 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a041bf-6dae-4b00-9fb7-30b07b16bdbd-util" (OuterVolumeSpecName: "util") pod "b6a041bf-6dae-4b00-9fb7-30b07b16bdbd" (UID: "b6a041bf-6dae-4b00-9fb7-30b07b16bdbd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:27:00.894099 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:00.894068 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn" Jul 21 00:27:00.968292 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:00.968249 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d48e193a-a228-4bfa-8bb0-b756f456bf10-bundle\") pod \"d48e193a-a228-4bfa-8bb0-b756f456bf10\" (UID: \"d48e193a-a228-4bfa-8bb0-b756f456bf10\") " Jul 21 00:27:00.968453 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:00.968309 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d48e193a-a228-4bfa-8bb0-b756f456bf10-util\") pod \"d48e193a-a228-4bfa-8bb0-b756f456bf10\" (UID: \"d48e193a-a228-4bfa-8bb0-b756f456bf10\") " Jul 21 00:27:00.968453 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:00.968367 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q696k\" (UniqueName: \"kubernetes.io/projected/d48e193a-a228-4bfa-8bb0-b756f456bf10-kube-api-access-q696k\") pod \"d48e193a-a228-4bfa-8bb0-b756f456bf10\" (UID: \"d48e193a-a228-4bfa-8bb0-b756f456bf10\") " Jul 21 00:27:00.968557 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:00.968533 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b6a041bf-6dae-4b00-9fb7-30b07b16bdbd-bundle\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:27:00.968557 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:00.968546 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhqdd\" (UniqueName: \"kubernetes.io/projected/b6a041bf-6dae-4b00-9fb7-30b07b16bdbd-kube-api-access-dhqdd\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:27:00.968557 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:00.968556 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b6a041bf-6dae-4b00-9fb7-30b07b16bdbd-util\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:27:00.968911 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:00.968884 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48e193a-a228-4bfa-8bb0-b756f456bf10-bundle" (OuterVolumeSpecName: "bundle") pod "d48e193a-a228-4bfa-8bb0-b756f456bf10" (UID: "d48e193a-a228-4bfa-8bb0-b756f456bf10"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:27:00.970542 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:00.970514 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48e193a-a228-4bfa-8bb0-b756f456bf10-kube-api-access-q696k" (OuterVolumeSpecName: "kube-api-access-q696k") pod "d48e193a-a228-4bfa-8bb0-b756f456bf10" (UID: "d48e193a-a228-4bfa-8bb0-b756f456bf10"). InnerVolumeSpecName "kube-api-access-q696k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:27:00.974028 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:00.974005 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48e193a-a228-4bfa-8bb0-b756f456bf10-util" (OuterVolumeSpecName: "util") pod "d48e193a-a228-4bfa-8bb0-b756f456bf10" (UID: "d48e193a-a228-4bfa-8bb0-b756f456bf10"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:27:01.069676 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:01.069641 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d48e193a-a228-4bfa-8bb0-b756f456bf10-util\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:27:01.069676 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:01.069670 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q696k\" (UniqueName: \"kubernetes.io/projected/d48e193a-a228-4bfa-8bb0-b756f456bf10-kube-api-access-q696k\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:27:01.069676 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:01.069681 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d48e193a-a228-4bfa-8bb0-b756f456bf10-bundle\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:27:01.640446 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:01.640419 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn" Jul 21 00:27:01.640446 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:01.640435 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn" event={"ID":"d48e193a-a228-4bfa-8bb0-b756f456bf10","Type":"ContainerDied","Data":"33521aa241a525b99364ca5140c7b61237350d62758c35a02a05df75f76bfd8c"} Jul 21 00:27:01.640973 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:01.640478 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33521aa241a525b99364ca5140c7b61237350d62758c35a02a05df75f76bfd8c" Jul 21 00:27:01.642511 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:01.642480 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms" event={"ID":"b6a041bf-6dae-4b00-9fb7-30b07b16bdbd","Type":"ContainerDied","Data":"1df133051264cc26ffaa8a5712e3806a56ca66ac4e2255a526d13b62f205eaeb"} Jul 21 00:27:01.642511 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:01.642503 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms" Jul 21 00:27:01.642751 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:01.642522 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1df133051264cc26ffaa8a5712e3806a56ca66ac4e2255a526d13b62f205eaeb" Jul 21 00:27:01.644646 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:01.644618 2580 generic.go:356] "Generic (PLEG): container finished" podID="1f66e031-60a1-4f6d-942b-782770b2ea3d" containerID="86b86a94ec4c8441737d3ecaff534634e9c464f706ec0a93108f5f940770a47f" exitCode=0 Jul 21 00:27:01.644770 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:01.644673 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7" event={"ID":"1f66e031-60a1-4f6d-942b-782770b2ea3d","Type":"ContainerDied","Data":"86b86a94ec4c8441737d3ecaff534634e9c464f706ec0a93108f5f940770a47f"} Jul 21 00:27:01.769051 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:01.769024 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl" Jul 21 00:27:01.876482 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:01.876446 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd296917-fa8f-471e-b4fc-9795c3be221d-util\") pod \"dd296917-fa8f-471e-b4fc-9795c3be221d\" (UID: \"dd296917-fa8f-471e-b4fc-9795c3be221d\") " Jul 21 00:27:01.876698 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:01.876494 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd296917-fa8f-471e-b4fc-9795c3be221d-bundle\") pod \"dd296917-fa8f-471e-b4fc-9795c3be221d\" (UID: \"dd296917-fa8f-471e-b4fc-9795c3be221d\") " Jul 21 00:27:01.876698 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:01.876545 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ps4zq\" (UniqueName: \"kubernetes.io/projected/dd296917-fa8f-471e-b4fc-9795c3be221d-kube-api-access-ps4zq\") pod \"dd296917-fa8f-471e-b4fc-9795c3be221d\" (UID: \"dd296917-fa8f-471e-b4fc-9795c3be221d\") " Jul 21 00:27:01.877046 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:01.877022 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd296917-fa8f-471e-b4fc-9795c3be221d-bundle" (OuterVolumeSpecName: "bundle") pod "dd296917-fa8f-471e-b4fc-9795c3be221d" (UID: "dd296917-fa8f-471e-b4fc-9795c3be221d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:27:01.878969 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:01.878942 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd296917-fa8f-471e-b4fc-9795c3be221d-kube-api-access-ps4zq" (OuterVolumeSpecName: "kube-api-access-ps4zq") pod "dd296917-fa8f-471e-b4fc-9795c3be221d" (UID: "dd296917-fa8f-471e-b4fc-9795c3be221d"). InnerVolumeSpecName "kube-api-access-ps4zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:27:01.971761 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:01.971660 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd296917-fa8f-471e-b4fc-9795c3be221d-util" (OuterVolumeSpecName: "util") pod "dd296917-fa8f-471e-b4fc-9795c3be221d" (UID: "dd296917-fa8f-471e-b4fc-9795c3be221d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:27:01.978000 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:01.977971 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd296917-fa8f-471e-b4fc-9795c3be221d-util\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:27:01.978000 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:01.977999 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd296917-fa8f-471e-b4fc-9795c3be221d-bundle\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:27:01.978000 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:01.978008 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps4zq\" (UniqueName: \"kubernetes.io/projected/dd296917-fa8f-471e-b4fc-9795c3be221d-kube-api-access-ps4zq\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:27:02.651436 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:02.651403 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl" Jul 21 00:27:02.651823 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:02.651436 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl" event={"ID":"dd296917-fa8f-471e-b4fc-9795c3be221d","Type":"ContainerDied","Data":"357636c76216466f614a0a36dff3bd8187a550ae9287d2d2e31e0a55be254dd1"} Jul 21 00:27:02.651823 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:02.651483 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="357636c76216466f614a0a36dff3bd8187a550ae9287d2d2e31e0a55be254dd1" Jul 21 00:27:02.783579 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:02.783547 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7" Jul 21 00:27:02.887749 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:02.887709 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-962kc\" (UniqueName: \"kubernetes.io/projected/1f66e031-60a1-4f6d-942b-782770b2ea3d-kube-api-access-962kc\") pod \"1f66e031-60a1-4f6d-942b-782770b2ea3d\" (UID: \"1f66e031-60a1-4f6d-942b-782770b2ea3d\") " Jul 21 00:27:02.887900 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:02.887813 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f66e031-60a1-4f6d-942b-782770b2ea3d-util\") pod \"1f66e031-60a1-4f6d-942b-782770b2ea3d\" (UID: \"1f66e031-60a1-4f6d-942b-782770b2ea3d\") " Jul 21 00:27:02.887954 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:02.887909 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f66e031-60a1-4f6d-942b-782770b2ea3d-bundle\") pod \"1f66e031-60a1-4f6d-942b-782770b2ea3d\" (UID: \"1f66e031-60a1-4f6d-942b-782770b2ea3d\") " Jul 21 00:27:02.888396 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:02.888375 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f66e031-60a1-4f6d-942b-782770b2ea3d-bundle" (OuterVolumeSpecName: "bundle") pod "1f66e031-60a1-4f6d-942b-782770b2ea3d" (UID: "1f66e031-60a1-4f6d-942b-782770b2ea3d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:27:02.890201 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:02.890170 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f66e031-60a1-4f6d-942b-782770b2ea3d-kube-api-access-962kc" (OuterVolumeSpecName: "kube-api-access-962kc") pod "1f66e031-60a1-4f6d-942b-782770b2ea3d" (UID: "1f66e031-60a1-4f6d-942b-782770b2ea3d"). InnerVolumeSpecName "kube-api-access-962kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:27:02.989113 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:02.989017 2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f66e031-60a1-4f6d-942b-782770b2ea3d-bundle\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:27:02.989113 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:02.989056 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-962kc\" (UniqueName: \"kubernetes.io/projected/1f66e031-60a1-4f6d-942b-782770b2ea3d-kube-api-access-962kc\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:27:03.011170 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:03.011136 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f66e031-60a1-4f6d-942b-782770b2ea3d-util" (OuterVolumeSpecName: "util") pod "1f66e031-60a1-4f6d-942b-782770b2ea3d" (UID: "1f66e031-60a1-4f6d-942b-782770b2ea3d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:27:03.089851 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:03.089814 2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f66e031-60a1-4f6d-942b-782770b2ea3d-util\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:27:03.665341 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:03.665297 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7" event={"ID":"1f66e031-60a1-4f6d-942b-782770b2ea3d","Type":"ContainerDied","Data":"89a96dec8f3b37c1a0c5c7654aa4352c2673601a16d7976328ffe006dc86ca88"} Jul 21 00:27:03.665768 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:03.665345 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7" Jul 21 00:27:03.665768 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:03.665379 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89a96dec8f3b37c1a0c5c7654aa4352c2673601a16d7976328ffe006dc86ca88" Jul 21 00:27:07.048931 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:07.048836 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:27:07.048931 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:07.048902 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:27:07.053844 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:07.053820 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:27:07.690868 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:07.690837 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5944955d98-bhzfc" Jul 21 00:27:07.738059 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:07.738018 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68df77886f-ghr8l"] Jul 21 00:27:07.852901 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:07.852865 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:07.960952 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:07.960860 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:08.055692 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:08.055655 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:08.124248 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:08.124211 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:08.693734 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:08.693701 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:12.118002 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:12.117972 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:13.419622 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.419571 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb"] Jul 21 00:27:13.420083 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420055 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6a041bf-6dae-4b00-9fb7-30b07b16bdbd" containerName="extract" Jul 21 00:27:13.420083 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420073 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6a041bf-6dae-4b00-9fb7-30b07b16bdbd" containerName="extract" Jul 21 00:27:13.420083 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420081 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd296917-fa8f-471e-b4fc-9795c3be221d" containerName="extract" Jul 21 00:27:13.420306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420088 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd296917-fa8f-471e-b4fc-9795c3be221d" containerName="extract" Jul 21 00:27:13.420306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420099 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f66e031-60a1-4f6d-942b-782770b2ea3d" containerName="extract" Jul 21 00:27:13.420306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420108 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f66e031-60a1-4f6d-942b-782770b2ea3d" containerName="extract" Jul 21 00:27:13.420306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420117 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d48e193a-a228-4bfa-8bb0-b756f456bf10" containerName="util" Jul 21 00:27:13.420306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420124 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d48e193a-a228-4bfa-8bb0-b756f456bf10" containerName="util" Jul 21 00:27:13.420306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420133 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f66e031-60a1-4f6d-942b-782770b2ea3d" containerName="util" Jul 21 00:27:13.420306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420141 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f66e031-60a1-4f6d-942b-782770b2ea3d" containerName="util" Jul 21 00:27:13.420306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420157 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d48e193a-a228-4bfa-8bb0-b756f456bf10" containerName="extract" Jul 21 00:27:13.420306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420164 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d48e193a-a228-4bfa-8bb0-b756f456bf10" containerName="extract" Jul 21 00:27:13.420306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420179 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6a041bf-6dae-4b00-9fb7-30b07b16bdbd" containerName="util" Jul 21 00:27:13.420306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420188 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6a041bf-6dae-4b00-9fb7-30b07b16bdbd" containerName="util" Jul 21 00:27:13.420306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420196 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6a041bf-6dae-4b00-9fb7-30b07b16bdbd" containerName="pull" Jul 21 00:27:13.420306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420204 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6a041bf-6dae-4b00-9fb7-30b07b16bdbd" containerName="pull" Jul 21 00:27:13.420306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420217 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd296917-fa8f-471e-b4fc-9795c3be221d" containerName="pull" Jul 21 00:27:13.420306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420225 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd296917-fa8f-471e-b4fc-9795c3be221d" containerName="pull" Jul 21 00:27:13.420306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420233 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d48e193a-a228-4bfa-8bb0-b756f456bf10" containerName="pull" Jul 21 00:27:13.420306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420242 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d48e193a-a228-4bfa-8bb0-b756f456bf10" containerName="pull" Jul 21 00:27:13.420306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420253 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd296917-fa8f-471e-b4fc-9795c3be221d" containerName="util" Jul 21 00:27:13.420306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420259 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd296917-fa8f-471e-b4fc-9795c3be221d" containerName="util" Jul 21 00:27:13.420306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420268 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f66e031-60a1-4f6d-942b-782770b2ea3d" containerName="pull" Jul 21 00:27:13.420306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420276 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f66e031-60a1-4f6d-942b-782770b2ea3d" containerName="pull" Jul 21 00:27:13.421215 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420375 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f66e031-60a1-4f6d-942b-782770b2ea3d" containerName="extract" Jul 21 00:27:13.421215 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420387 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d48e193a-a228-4bfa-8bb0-b756f456bf10" containerName="extract" Jul 21 00:27:13.421215 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420400 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6a041bf-6dae-4b00-9fb7-30b07b16bdbd" containerName="extract" Jul 21 00:27:13.421215 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.420410 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd296917-fa8f-471e-b4fc-9795c3be221d" containerName="extract" Jul 21 00:27:13.426564 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.426538 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" Jul 21 00:27:13.429249 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.429225 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-btcj9\"" Jul 21 00:27:13.437708 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.437674 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb"] Jul 21 00:27:13.583079 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.583037 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n5tw\" (UniqueName: \"kubernetes.io/projected/704c12fe-ac99-4d4e-84e3-091159c46428-kube-api-access-6n5tw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-k5rkb\" (UID: \"704c12fe-ac99-4d4e-84e3-091159c46428\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" Jul 21 00:27:13.583261 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.583097 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/704c12fe-ac99-4d4e-84e3-091159c46428-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-k5rkb\" (UID: \"704c12fe-ac99-4d4e-84e3-091159c46428\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" Jul 21 00:27:13.684155 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.684057 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n5tw\" (UniqueName: \"kubernetes.io/projected/704c12fe-ac99-4d4e-84e3-091159c46428-kube-api-access-6n5tw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-k5rkb\" (UID: \"704c12fe-ac99-4d4e-84e3-091159c46428\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" Jul 21 00:27:13.684155 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.684115 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/704c12fe-ac99-4d4e-84e3-091159c46428-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-k5rkb\" (UID: \"704c12fe-ac99-4d4e-84e3-091159c46428\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" Jul 21 00:27:13.684468 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.684451 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/704c12fe-ac99-4d4e-84e3-091159c46428-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-k5rkb\" (UID: \"704c12fe-ac99-4d4e-84e3-091159c46428\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" Jul 21 00:27:13.686898 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.686866 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:13.692011 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.691984 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n5tw\" (UniqueName: \"kubernetes.io/projected/704c12fe-ac99-4d4e-84e3-091159c46428-kube-api-access-6n5tw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-k5rkb\" (UID: \"704c12fe-ac99-4d4e-84e3-091159c46428\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" Jul 21 00:27:13.737546 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.737507 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" Jul 21 00:27:13.881196 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:13.881166 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb"] Jul 21 00:27:13.883394 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:27:13.883366 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704c12fe_ac99_4d4e_84e3_091159c46428.slice/crio-db0a82a49537088214c570dbf4e41a312ddccdb3310c78eb61a77a61dc1d12a3 WatchSource:0}: Error finding container db0a82a49537088214c570dbf4e41a312ddccdb3310c78eb61a77a61dc1d12a3: Status 404 returned error can't find the container with id db0a82a49537088214c570dbf4e41a312ddccdb3310c78eb61a77a61dc1d12a3 Jul 21 00:27:14.084777 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:14.084740 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:14.724420 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:14.724376 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" event={"ID":"704c12fe-ac99-4d4e-84e3-091159c46428","Type":"ContainerStarted","Data":"db0a82a49537088214c570dbf4e41a312ddccdb3310c78eb61a77a61dc1d12a3"} Jul 21 00:27:15.138805 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:15.138773 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:15.245521 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:15.245475 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:16.025150 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:16.025114 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-s8c42"] Jul 21 00:27:16.028993 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:16.028973 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-s8c42" Jul 21 00:27:16.031400 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:16.031377 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-4r7j7\"" Jul 21 00:27:16.040970 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:16.040938 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-s8c42"] Jul 21 00:27:16.107096 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:16.107050 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84ppk\" (UniqueName: \"kubernetes.io/projected/43e79b6a-729d-4a56-96c1-10d0e6e6f935-kube-api-access-84ppk\") pod \"authorino-operator-75cc8d5cb7-s8c42\" (UID: \"43e79b6a-729d-4a56-96c1-10d0e6e6f935\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-s8c42" Jul 21 00:27:16.208256 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:16.208221 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84ppk\" (UniqueName: \"kubernetes.io/projected/43e79b6a-729d-4a56-96c1-10d0e6e6f935-kube-api-access-84ppk\") pod \"authorino-operator-75cc8d5cb7-s8c42\" (UID: \"43e79b6a-729d-4a56-96c1-10d0e6e6f935\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-s8c42" Jul 21 00:27:16.223341 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:16.223304 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84ppk\" (UniqueName: \"kubernetes.io/projected/43e79b6a-729d-4a56-96c1-10d0e6e6f935-kube-api-access-84ppk\") pod \"authorino-operator-75cc8d5cb7-s8c42\" (UID: \"43e79b6a-729d-4a56-96c1-10d0e6e6f935\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-s8c42" Jul 21 00:27:16.340214 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:16.340184 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-s8c42" Jul 21 00:27:16.489076 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:16.489051 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-s8c42"] Jul 21 00:27:16.490750 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:27:16.490716 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e79b6a_729d_4a56_96c1_10d0e6e6f935.slice/crio-0a93384fdf56da5c00c14a799bd6913ab0b4f39a22cadcd7b65c4cb31f0191bc WatchSource:0}: Error finding container 0a93384fdf56da5c00c14a799bd6913ab0b4f39a22cadcd7b65c4cb31f0191bc: Status 404 returned error can't find the container with id 0a93384fdf56da5c00c14a799bd6913ab0b4f39a22cadcd7b65c4cb31f0191bc Jul 21 00:27:16.716833 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:16.716751 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:16.739120 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:16.739082 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-s8c42" event={"ID":"43e79b6a-729d-4a56-96c1-10d0e6e6f935","Type":"ContainerStarted","Data":"0a93384fdf56da5c00c14a799bd6913ab0b4f39a22cadcd7b65c4cb31f0191bc"} Jul 21 00:27:16.849229 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:16.849195 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:18.369270 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:18.369169 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-27jbj"] Jul 21 00:27:18.380076 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:18.380043 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-27jbj" Jul 21 00:27:18.383152 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:18.383123 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-xtfqf\"" Jul 21 00:27:18.383303 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:18.383202 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 21 00:27:18.392951 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:18.392924 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-27jbj"] Jul 21 00:27:18.532273 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:18.532230 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkb67\" (UniqueName: \"kubernetes.io/projected/1705926d-8894-488d-add2-121845ea0ff7-kube-api-access-jkb67\") pod \"dns-operator-controller-manager-864dd78c4c-27jbj\" (UID: \"1705926d-8894-488d-add2-121845ea0ff7\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-27jbj" Jul 21 00:27:18.633676 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:18.633568 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkb67\" (UniqueName: \"kubernetes.io/projected/1705926d-8894-488d-add2-121845ea0ff7-kube-api-access-jkb67\") pod \"dns-operator-controller-manager-864dd78c4c-27jbj\" (UID: \"1705926d-8894-488d-add2-121845ea0ff7\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-27jbj" Jul 21 00:27:18.648619 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:18.648572 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkb67\" (UniqueName: \"kubernetes.io/projected/1705926d-8894-488d-add2-121845ea0ff7-kube-api-access-jkb67\") pod \"dns-operator-controller-manager-864dd78c4c-27jbj\" (UID: \"1705926d-8894-488d-add2-121845ea0ff7\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-27jbj" Jul 21 00:27:18.684061 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:18.684023 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:18.698914 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:18.698880 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-27jbj" Jul 21 00:27:18.780183 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:18.780140 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:18.899672 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:18.899557 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:19.100227 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:27:19.100193 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1705926d_8894_488d_add2_121845ea0ff7.slice/crio-4b78dacfa9b5bbfde7153625433182693768bedd50b47c8f578b4d189a402fc8 WatchSource:0}: Error finding container 4b78dacfa9b5bbfde7153625433182693768bedd50b47c8f578b4d189a402fc8: Status 404 returned error can't find the container with id 4b78dacfa9b5bbfde7153625433182693768bedd50b47c8f578b4d189a402fc8 Jul 21 00:27:19.101153 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:19.101126 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-27jbj"] Jul 21 00:27:19.381307 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:19.381269 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:19.483092 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:19.483059 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:19.760220 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:19.760121 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-s8c42" event={"ID":"43e79b6a-729d-4a56-96c1-10d0e6e6f935","Type":"ContainerStarted","Data":"1d027a91600cf5f8a8e898365a7d28bb04f805bdf2e998ea4d80ff118aa2f8c5"} Jul 21 00:27:19.760395 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:19.760249 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-s8c42" Jul 21 00:27:19.761621 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:19.761581 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-27jbj" event={"ID":"1705926d-8894-488d-add2-121845ea0ff7","Type":"ContainerStarted","Data":"4b78dacfa9b5bbfde7153625433182693768bedd50b47c8f578b4d189a402fc8"} Jul 21 00:27:19.778413 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:19.778354 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-s8c42" podStartSLOduration=1.279027718 podStartE2EDuration="3.778334079s" totalImagesPullingTime="2.499306361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:27:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:27:19.775898161 +0000 UTC m=+633.991278482" watchObservedRunningTime="2026-07-21 00:27:19.778334079 +0000 UTC m=+633.993714404" Jul 21 00:27:20.651689 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:20.651647 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz"] Jul 21 00:27:20.655308 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:20.655282 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz" Jul 21 00:27:20.657876 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:20.657853 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-sn5ln\"" Jul 21 00:27:20.665874 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:20.665843 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz"] Jul 21 00:27:20.736905 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:20.736852 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:20.754062 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:20.754020 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v9rv\" (UniqueName: \"kubernetes.io/projected/1024828c-610d-47eb-9857-c9d99ffdd05b-kube-api-access-9v9rv\") pod \"limitador-operator-controller-manager-859d7b78bd-8s4tz\" (UID: \"1024828c-610d-47eb-9857-c9d99ffdd05b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz" Jul 21 00:27:20.855479 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:20.855451 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v9rv\" (UniqueName: \"kubernetes.io/projected/1024828c-610d-47eb-9857-c9d99ffdd05b-kube-api-access-9v9rv\") pod \"limitador-operator-controller-manager-859d7b78bd-8s4tz\" (UID: \"1024828c-610d-47eb-9857-c9d99ffdd05b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz" Jul 21 00:27:20.866431 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:20.866402 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v9rv\" (UniqueName: \"kubernetes.io/projected/1024828c-610d-47eb-9857-c9d99ffdd05b-kube-api-access-9v9rv\") pod \"limitador-operator-controller-manager-859d7b78bd-8s4tz\" (UID: \"1024828c-610d-47eb-9857-c9d99ffdd05b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz" Jul 21 00:27:20.968396 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:20.968364 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz" Jul 21 00:27:21.330687 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:21.330658 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz"] Jul 21 00:27:21.365694 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:27:21.365655 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1024828c_610d_47eb_9857_c9d99ffdd05b.slice/crio-3cc5061793e50ee65ab8e71dfc9ae71be8375512419d9178fc2410d8ff6fff71 WatchSource:0}: Error finding container 3cc5061793e50ee65ab8e71dfc9ae71be8375512419d9178fc2410d8ff6fff71: Status 404 returned error can't find the container with id 3cc5061793e50ee65ab8e71dfc9ae71be8375512419d9178fc2410d8ff6fff71 Jul 21 00:27:21.773416 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:21.773379 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz" event={"ID":"1024828c-610d-47eb-9857-c9d99ffdd05b","Type":"ContainerStarted","Data":"3cc5061793e50ee65ab8e71dfc9ae71be8375512419d9178fc2410d8ff6fff71"} Jul 21 00:27:21.775138 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:21.775104 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" event={"ID":"704c12fe-ac99-4d4e-84e3-091159c46428","Type":"ContainerStarted","Data":"607311a6707cd06b252675349c46431a4f48f9ed138f3ce5d18a9533fd9d3fc8"} Jul 21 00:27:21.775286 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:21.775212 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" Jul 21 00:27:21.796216 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:21.796164 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" podStartSLOduration=1.846747407 podStartE2EDuration="8.796147272s" totalImagesPullingTime="6.949399865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:27:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:27:21.793477641 +0000 UTC m=+636.008857977" watchObservedRunningTime="2026-07-21 00:27:21.796147272 +0000 UTC m=+636.011527596" Jul 21 00:27:22.490529 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:22.490483 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:22.551888 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:22.551855 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:23.788715 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:23.788675 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz" event={"ID":"1024828c-610d-47eb-9857-c9d99ffdd05b","Type":"ContainerStarted","Data":"c77846c6c9ae05a3a4568a94538d54a03e81952f321e71e36a89c06b72081fc2"} Jul 21 00:27:23.789214 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:23.788796 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz" Jul 21 00:27:23.790634 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:23.790590 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-27jbj" event={"ID":"1705926d-8894-488d-add2-121845ea0ff7","Type":"ContainerStarted","Data":"d2b15507ad288f70292aef10c46932a1857cfd1714fe8f6e3b1e4c5be7b65b79"} Jul 21 00:27:23.790737 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:23.790656 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-27jbj" Jul 21 00:27:23.809270 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:23.809219 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz" podStartSLOduration=2.109181771 podStartE2EDuration="3.809201766s" totalImagesPullingTime="1.700019995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:27:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:27:23.807155835 +0000 UTC m=+638.022536160" watchObservedRunningTime="2026-07-21 00:27:23.809201766 +0000 UTC m=+638.024582091" Jul 21 00:27:23.834060 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:23.834008 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-27jbj" podStartSLOduration=2.260812263 podStartE2EDuration="5.833989876s" totalImagesPullingTime="3.573177613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:27:23.831592172 +0000 UTC m=+638.046972505" watchObservedRunningTime="2026-07-21 00:27:23.833989876 +0000 UTC m=+638.049370200" Jul 21 00:27:23.888964 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:23.888913 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:24.247003 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:24.246965 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:24.347479 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:24.347439 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:24.447706 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:24.447665 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:30.769274 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:30.769238 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-s8c42" Jul 21 00:27:32.759361 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:32.759316 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68df77886f-ghr8l" podUID="b8f2e8f7-b292-4100-af63-85dd51625ac5" containerName="console" containerID="cri-o://b115e29434d08e0b69d303f4162be4fd5816d724aab3f28b04b74545902c616e" gracePeriod=15 Jul 21 00:27:32.783882 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:32.783850 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" Jul 21 00:27:32.798855 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:32.798825 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:32.911157 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:32.911128 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:33.015866 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.015794 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:27:33.163323 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.163287 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8f2e8f7-b292-4100-af63-85dd51625ac5-console-config\") pod \"b8f2e8f7-b292-4100-af63-85dd51625ac5\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " Jul 21 00:27:33.163527 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.163341 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8f2e8f7-b292-4100-af63-85dd51625ac5-trusted-ca-bundle\") pod \"b8f2e8f7-b292-4100-af63-85dd51625ac5\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " Jul 21 00:27:33.163527 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.163362 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8f2e8f7-b292-4100-af63-85dd51625ac5-oauth-serving-cert\") pod \"b8f2e8f7-b292-4100-af63-85dd51625ac5\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " Jul 21 00:27:33.163527 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.163398 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8f2e8f7-b292-4100-af63-85dd51625ac5-console-oauth-config\") pod \"b8f2e8f7-b292-4100-af63-85dd51625ac5\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " Jul 21 00:27:33.163527 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.163435 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8f2e8f7-b292-4100-af63-85dd51625ac5-console-serving-cert\") pod \"b8f2e8f7-b292-4100-af63-85dd51625ac5\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " Jul 21 00:27:33.163527 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.163465 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8f2e8f7-b292-4100-af63-85dd51625ac5-service-ca\") pod \"b8f2e8f7-b292-4100-af63-85dd51625ac5\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " Jul 21 00:27:33.163818 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.163536 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cm878\" (UniqueName: \"kubernetes.io/projected/b8f2e8f7-b292-4100-af63-85dd51625ac5-kube-api-access-cm878\") pod \"b8f2e8f7-b292-4100-af63-85dd51625ac5\" (UID: \"b8f2e8f7-b292-4100-af63-85dd51625ac5\") " Jul 21 00:27:33.163818 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.163793 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8f2e8f7-b292-4100-af63-85dd51625ac5-console-config" (OuterVolumeSpecName: "console-config") pod "b8f2e8f7-b292-4100-af63-85dd51625ac5" (UID: "b8f2e8f7-b292-4100-af63-85dd51625ac5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:27:33.163918 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.163820 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8f2e8f7-b292-4100-af63-85dd51625ac5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b8f2e8f7-b292-4100-af63-85dd51625ac5" (UID: "b8f2e8f7-b292-4100-af63-85dd51625ac5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:27:33.164102 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.164077 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8f2e8f7-b292-4100-af63-85dd51625ac5-service-ca" (OuterVolumeSpecName: "service-ca") pod "b8f2e8f7-b292-4100-af63-85dd51625ac5" (UID: "b8f2e8f7-b292-4100-af63-85dd51625ac5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:27:33.164102 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.164085 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8f2e8f7-b292-4100-af63-85dd51625ac5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b8f2e8f7-b292-4100-af63-85dd51625ac5" (UID: "b8f2e8f7-b292-4100-af63-85dd51625ac5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:27:33.165872 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.165838 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8f2e8f7-b292-4100-af63-85dd51625ac5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b8f2e8f7-b292-4100-af63-85dd51625ac5" (UID: "b8f2e8f7-b292-4100-af63-85dd51625ac5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:27:33.166011 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.165989 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8f2e8f7-b292-4100-af63-85dd51625ac5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b8f2e8f7-b292-4100-af63-85dd51625ac5" (UID: "b8f2e8f7-b292-4100-af63-85dd51625ac5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:27:33.166058 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.166025 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f2e8f7-b292-4100-af63-85dd51625ac5-kube-api-access-cm878" (OuterVolumeSpecName: "kube-api-access-cm878") pod "b8f2e8f7-b292-4100-af63-85dd51625ac5" (UID: "b8f2e8f7-b292-4100-af63-85dd51625ac5"). InnerVolumeSpecName "kube-api-access-cm878". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:27:33.265108 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.265060 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8f2e8f7-b292-4100-af63-85dd51625ac5-console-config\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:27:33.265108 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.265099 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8f2e8f7-b292-4100-af63-85dd51625ac5-trusted-ca-bundle\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:27:33.265108 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.265109 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8f2e8f7-b292-4100-af63-85dd51625ac5-oauth-serving-cert\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:27:33.265108 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.265117 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8f2e8f7-b292-4100-af63-85dd51625ac5-console-oauth-config\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:27:33.265391 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.265126 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8f2e8f7-b292-4100-af63-85dd51625ac5-console-serving-cert\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:27:33.265391 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.265134 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8f2e8f7-b292-4100-af63-85dd51625ac5-service-ca\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:27:33.265391 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.265142 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm878\" (UniqueName: \"kubernetes.io/projected/b8f2e8f7-b292-4100-af63-85dd51625ac5-kube-api-access-cm878\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:27:33.849445 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.849411 2580 generic.go:356] "Generic (PLEG): container finished" podID="b8f2e8f7-b292-4100-af63-85dd51625ac5" containerID="b115e29434d08e0b69d303f4162be4fd5816d724aab3f28b04b74545902c616e" exitCode=0 Jul 21 00:27:33.849860 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.849459 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68df77886f-ghr8l" Jul 21 00:27:33.849860 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.849461 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68df77886f-ghr8l" event={"ID":"b8f2e8f7-b292-4100-af63-85dd51625ac5","Type":"ContainerDied","Data":"b115e29434d08e0b69d303f4162be4fd5816d724aab3f28b04b74545902c616e"} Jul 21 00:27:33.849860 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.849581 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68df77886f-ghr8l" event={"ID":"b8f2e8f7-b292-4100-af63-85dd51625ac5","Type":"ContainerDied","Data":"a4aab7967035c014fe6573b5af41a0def38befd63af8f43faaeb9f331561096c"} Jul 21 00:27:33.849860 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.849632 2580 scope.go:117] "RemoveContainer" containerID="b115e29434d08e0b69d303f4162be4fd5816d724aab3f28b04b74545902c616e" Jul 21 00:27:33.859229 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.859210 2580 scope.go:117] "RemoveContainer" containerID="b115e29434d08e0b69d303f4162be4fd5816d724aab3f28b04b74545902c616e" Jul 21 00:27:33.859519 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:27:33.859497 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b115e29434d08e0b69d303f4162be4fd5816d724aab3f28b04b74545902c616e\": container with ID starting with b115e29434d08e0b69d303f4162be4fd5816d724aab3f28b04b74545902c616e not found: ID does not exist" containerID="b115e29434d08e0b69d303f4162be4fd5816d724aab3f28b04b74545902c616e" Jul 21 00:27:33.859565 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.859529 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b115e29434d08e0b69d303f4162be4fd5816d724aab3f28b04b74545902c616e"} err="failed to get container status \"b115e29434d08e0b69d303f4162be4fd5816d724aab3f28b04b74545902c616e\": rpc error: code = NotFound desc = could not find container \"b115e29434d08e0b69d303f4162be4fd5816d724aab3f28b04b74545902c616e\": container with ID starting with b115e29434d08e0b69d303f4162be4fd5816d724aab3f28b04b74545902c616e not found: ID does not exist" Jul 21 00:27:33.880261 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.880221 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68df77886f-ghr8l"] Jul 21 00:27:33.891100 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:33.891068 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68df77886f-ghr8l"] Jul 21 00:27:34.297712 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.297618 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8f2e8f7-b292-4100-af63-85dd51625ac5" path="/var/lib/kubelet/pods/b8f2e8f7-b292-4100-af63-85dd51625ac5/volumes" Jul 21 00:27:34.653971 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.653934 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb"] Jul 21 00:27:34.654171 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.654110 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" podUID="704c12fe-ac99-4d4e-84e3-091159c46428" containerName="manager" containerID="cri-o://607311a6707cd06b252675349c46431a4f48f9ed138f3ce5d18a9533fd9d3fc8" gracePeriod=2 Jul 21 00:27:34.664861 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.664835 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb"] Jul 21 00:27:34.667735 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:27:34.667692 2580 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"704c12fe-ac99-4d4e-84e3-091159c46428\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-k5rkb\": pods \"kuadrant-operator-controller-manager-8684c9ff49-k5rkb\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" Jul 21 00:27:34.688390 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.688349 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whtsp"] Jul 21 00:27:34.688941 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.688920 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="704c12fe-ac99-4d4e-84e3-091159c46428" containerName="manager" Jul 21 00:27:34.689038 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.688944 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="704c12fe-ac99-4d4e-84e3-091159c46428" containerName="manager" Jul 21 00:27:34.689038 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.688960 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8f2e8f7-b292-4100-af63-85dd51625ac5" containerName="console" Jul 21 00:27:34.689038 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.688969 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8f2e8f7-b292-4100-af63-85dd51625ac5" containerName="console" Jul 21 00:27:34.689151 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.689074 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="704c12fe-ac99-4d4e-84e3-091159c46428" containerName="manager" Jul 21 00:27:34.689151 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.689088 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8f2e8f7-b292-4100-af63-85dd51625ac5" containerName="console" Jul 21 00:27:34.692610 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.692575 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whtsp" Jul 21 00:27:34.708164 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.708134 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whtsp"] Jul 21 00:27:34.710237 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.710211 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz"] Jul 21 00:27:34.710446 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.710421 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz" podUID="1024828c-610d-47eb-9857-c9d99ffdd05b" containerName="manager" containerID="cri-o://c77846c6c9ae05a3a4568a94538d54a03e81952f321e71e36a89c06b72081fc2" gracePeriod=2 Jul 21 00:27:34.712404 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.712382 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz" Jul 21 00:27:34.721764 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:27:34.721674 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k5rkb\" is forbidden: User \"system:node:ip-10-0-139-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-117.ec2.internal' and this object" podUID="704c12fe-ac99-4d4e-84e3-091159c46428" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" Jul 21 00:27:34.726280 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:27:34.726130 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k5rkb\" is forbidden: User \"system:node:ip-10-0-139-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-117.ec2.internal' and this object" podUID="704c12fe-ac99-4d4e-84e3-091159c46428" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" Jul 21 00:27:34.728873 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.728845 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz"] Jul 21 00:27:34.729421 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:27:34.729396 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8s4tz\" is forbidden: User \"system:node:ip-10-0-139-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-117.ec2.internal' and this object" podUID="1024828c-610d-47eb-9857-c9d99ffdd05b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz" Jul 21 00:27:34.745816 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.745783 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rsfpd"] Jul 21 00:27:34.746174 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.746157 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1024828c-610d-47eb-9857-c9d99ffdd05b" containerName="manager" Jul 21 00:27:34.746174 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.746174 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1024828c-610d-47eb-9857-c9d99ffdd05b" containerName="manager" Jul 21 00:27:34.746346 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.746246 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1024828c-610d-47eb-9857-c9d99ffdd05b" containerName="manager" Jul 21 00:27:34.749260 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.749241 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rsfpd" Jul 21 00:27:34.761733 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.761703 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rsfpd"] Jul 21 00:27:34.776453 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.776422 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rlcb\" (UniqueName: \"kubernetes.io/projected/0fa997ae-55a5-49ac-833d-88e3c070f32c-kube-api-access-2rlcb\") pod \"limitador-operator-controller-manager-859d7b78bd-rsfpd\" (UID: \"0fa997ae-55a5-49ac-833d-88e3c070f32c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rsfpd" Jul 21 00:27:34.776670 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.776467 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8564d093-f96c-49e5-a972-7979677085f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-whtsp\" (UID: \"8564d093-f96c-49e5-a972-7979677085f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whtsp" Jul 21 00:27:34.776670 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.776550 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9x5m\" (UniqueName: \"kubernetes.io/projected/8564d093-f96c-49e5-a972-7979677085f3-kube-api-access-q9x5m\") pod \"kuadrant-operator-controller-manager-8684c9ff49-whtsp\" (UID: \"8564d093-f96c-49e5-a972-7979677085f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whtsp" Jul 21 00:27:34.778055 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:27:34.778014 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8s4tz\" is forbidden: User \"system:node:ip-10-0-139-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-117.ec2.internal' and this object" podUID="1024828c-610d-47eb-9857-c9d99ffdd05b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz" Jul 21 00:27:34.780728 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:27:34.780704 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k5rkb\" is forbidden: User \"system:node:ip-10-0-139-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-117.ec2.internal' and this object" podUID="704c12fe-ac99-4d4e-84e3-091159c46428" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" Jul 21 00:27:34.801141 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.801116 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-27jbj" Jul 21 00:27:34.803594 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:27:34.803566 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8s4tz\" is forbidden: User \"system:node:ip-10-0-139-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-117.ec2.internal' and this object" podUID="1024828c-610d-47eb-9857-c9d99ffdd05b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz" Jul 21 00:27:34.805690 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:27:34.805663 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k5rkb\" is forbidden: User \"system:node:ip-10-0-139-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-117.ec2.internal' and this object" podUID="704c12fe-ac99-4d4e-84e3-091159c46428" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" Jul 21 00:27:34.859566 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.859529 2580 generic.go:356] "Generic (PLEG): container finished" podID="1024828c-610d-47eb-9857-c9d99ffdd05b" containerID="c77846c6c9ae05a3a4568a94538d54a03e81952f321e71e36a89c06b72081fc2" exitCode=0 Jul 21 00:27:34.861430 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.861401 2580 generic.go:356] "Generic (PLEG): container finished" podID="704c12fe-ac99-4d4e-84e3-091159c46428" containerID="607311a6707cd06b252675349c46431a4f48f9ed138f3ce5d18a9533fd9d3fc8" exitCode=0 Jul 21 00:27:34.877682 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.877651 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9x5m\" (UniqueName: \"kubernetes.io/projected/8564d093-f96c-49e5-a972-7979677085f3-kube-api-access-q9x5m\") pod \"kuadrant-operator-controller-manager-8684c9ff49-whtsp\" (UID: \"8564d093-f96c-49e5-a972-7979677085f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whtsp" Jul 21 00:27:34.877866 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.877768 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rlcb\" (UniqueName: \"kubernetes.io/projected/0fa997ae-55a5-49ac-833d-88e3c070f32c-kube-api-access-2rlcb\") pod \"limitador-operator-controller-manager-859d7b78bd-rsfpd\" (UID: \"0fa997ae-55a5-49ac-833d-88e3c070f32c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rsfpd" Jul 21 00:27:34.877866 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.877825 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8564d093-f96c-49e5-a972-7979677085f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-whtsp\" (UID: \"8564d093-f96c-49e5-a972-7979677085f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whtsp" Jul 21 00:27:34.878189 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.878167 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8564d093-f96c-49e5-a972-7979677085f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-whtsp\" (UID: \"8564d093-f96c-49e5-a972-7979677085f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whtsp" Jul 21 00:27:34.890592 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.890556 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rlcb\" (UniqueName: \"kubernetes.io/projected/0fa997ae-55a5-49ac-833d-88e3c070f32c-kube-api-access-2rlcb\") pod \"limitador-operator-controller-manager-859d7b78bd-rsfpd\" (UID: \"0fa997ae-55a5-49ac-833d-88e3c070f32c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rsfpd" Jul 21 00:27:34.890769 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.890556 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9x5m\" (UniqueName: \"kubernetes.io/projected/8564d093-f96c-49e5-a972-7979677085f3-kube-api-access-q9x5m\") pod \"kuadrant-operator-controller-manager-8684c9ff49-whtsp\" (UID: \"8564d093-f96c-49e5-a972-7979677085f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whtsp" Jul 21 00:27:34.948183 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.948155 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" Jul 21 00:27:34.950966 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:27:34.950934 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8s4tz\" is forbidden: User \"system:node:ip-10-0-139-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-117.ec2.internal' and this object" podUID="1024828c-610d-47eb-9857-c9d99ffdd05b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz" Jul 21 00:27:34.953869 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:27:34.953841 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k5rkb\" is forbidden: User \"system:node:ip-10-0-139-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-117.ec2.internal' and this object" podUID="704c12fe-ac99-4d4e-84e3-091159c46428" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" Jul 21 00:27:34.955788 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:34.955773 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz" Jul 21 00:27:34.958060 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:27:34.958034 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8s4tz\" is forbidden: User \"system:node:ip-10-0-139-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-117.ec2.internal' and this object" podUID="1024828c-610d-47eb-9857-c9d99ffdd05b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz" Jul 21 00:27:34.960322 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:27:34.960299 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k5rkb\" is forbidden: User \"system:node:ip-10-0-139-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-117.ec2.internal' and this object" podUID="704c12fe-ac99-4d4e-84e3-091159c46428" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" Jul 21 00:27:35.079947 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.079915 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/704c12fe-ac99-4d4e-84e3-091159c46428-extensions-socket-volume\") pod \"704c12fe-ac99-4d4e-84e3-091159c46428\" (UID: \"704c12fe-ac99-4d4e-84e3-091159c46428\") " Jul 21 00:27:35.080131 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.079962 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n5tw\" (UniqueName: \"kubernetes.io/projected/704c12fe-ac99-4d4e-84e3-091159c46428-kube-api-access-6n5tw\") pod \"704c12fe-ac99-4d4e-84e3-091159c46428\" (UID: \"704c12fe-ac99-4d4e-84e3-091159c46428\") " Jul 21 00:27:35.080131 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.079987 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9v9rv\" (UniqueName: \"kubernetes.io/projected/1024828c-610d-47eb-9857-c9d99ffdd05b-kube-api-access-9v9rv\") pod \"1024828c-610d-47eb-9857-c9d99ffdd05b\" (UID: \"1024828c-610d-47eb-9857-c9d99ffdd05b\") " Jul 21 00:27:35.080503 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.080466 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/704c12fe-ac99-4d4e-84e3-091159c46428-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "704c12fe-ac99-4d4e-84e3-091159c46428" (UID: "704c12fe-ac99-4d4e-84e3-091159c46428"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:27:35.082163 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.082137 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704c12fe-ac99-4d4e-84e3-091159c46428-kube-api-access-6n5tw" (OuterVolumeSpecName: "kube-api-access-6n5tw") pod "704c12fe-ac99-4d4e-84e3-091159c46428" (UID: "704c12fe-ac99-4d4e-84e3-091159c46428"). InnerVolumeSpecName "kube-api-access-6n5tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:27:35.082163 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.082146 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1024828c-610d-47eb-9857-c9d99ffdd05b-kube-api-access-9v9rv" (OuterVolumeSpecName: "kube-api-access-9v9rv") pod "1024828c-610d-47eb-9857-c9d99ffdd05b" (UID: "1024828c-610d-47eb-9857-c9d99ffdd05b"). InnerVolumeSpecName "kube-api-access-9v9rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:27:35.108348 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.108307 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whtsp" Jul 21 00:27:35.114428 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.114403 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rsfpd" Jul 21 00:27:35.180575 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.180546 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n5tw\" (UniqueName: \"kubernetes.io/projected/704c12fe-ac99-4d4e-84e3-091159c46428-kube-api-access-6n5tw\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:27:35.180575 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.180571 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v9rv\" (UniqueName: \"kubernetes.io/projected/1024828c-610d-47eb-9857-c9d99ffdd05b-kube-api-access-9v9rv\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:27:35.180575 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.180583 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/704c12fe-ac99-4d4e-84e3-091159c46428-extensions-socket-volume\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:27:35.248302 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.248262 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:35.261106 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.261081 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whtsp"] Jul 21 00:27:35.263107 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:27:35.263074 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8564d093_f96c_49e5_a972_7979677085f3.slice/crio-ff8e8d86ccd3db1721bdf35323c9d85ecfe615893d8471e881fc2835b1157112 WatchSource:0}: Error finding container ff8e8d86ccd3db1721bdf35323c9d85ecfe615893d8471e881fc2835b1157112: Status 404 returned error can't find the container with id ff8e8d86ccd3db1721bdf35323c9d85ecfe615893d8471e881fc2835b1157112 Jul 21 00:27:35.286335 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.286283 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rsfpd"] Jul 21 00:27:35.288639 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:27:35.288592 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa997ae_55a5_49ac_833d_88e3c070f32c.slice/crio-eb62e7fb0c1aa89ae6f0564f578604ae0830c968da73d8e0783b04232d9d4ef9 WatchSource:0}: Error finding container eb62e7fb0c1aa89ae6f0564f578604ae0830c968da73d8e0783b04232d9d4ef9: Status 404 returned error can't find the container with id eb62e7fb0c1aa89ae6f0564f578604ae0830c968da73d8e0783b04232d9d4ef9 Jul 21 00:27:35.365212 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.365181 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:35.870241 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.870207 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whtsp" event={"ID":"8564d093-f96c-49e5-a972-7979677085f3","Type":"ContainerStarted","Data":"70f5590a55c83afe9c89399462f5dea8f6e47b2b288ac625044af8caf8505276"} Jul 21 00:27:35.870710 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.870258 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whtsp" event={"ID":"8564d093-f96c-49e5-a972-7979677085f3","Type":"ContainerStarted","Data":"ff8e8d86ccd3db1721bdf35323c9d85ecfe615893d8471e881fc2835b1157112"} Jul 21 00:27:35.870710 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.870307 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whtsp" Jul 21 00:27:35.871547 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.871533 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz" Jul 21 00:27:35.871647 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.871554 2580 scope.go:117] "RemoveContainer" containerID="c77846c6c9ae05a3a4568a94538d54a03e81952f321e71e36a89c06b72081fc2" Jul 21 00:27:35.872629 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:27:35.872581 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k5rkb\" is forbidden: User \"system:node:ip-10-0-139-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-117.ec2.internal' and this object" podUID="704c12fe-ac99-4d4e-84e3-091159c46428" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" Jul 21 00:27:35.873059 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.873037 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" Jul 21 00:27:35.875018 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.874993 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rsfpd" event={"ID":"0fa997ae-55a5-49ac-833d-88e3c070f32c","Type":"ContainerStarted","Data":"e0b315c9ef9cdae0d6b7f32f6a290646dc1769a7da325668b2ac3453ccf40579"} Jul 21 00:27:35.875142 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.875041 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rsfpd" event={"ID":"0fa997ae-55a5-49ac-833d-88e3c070f32c","Type":"ContainerStarted","Data":"eb62e7fb0c1aa89ae6f0564f578604ae0830c968da73d8e0783b04232d9d4ef9"} Jul 21 00:27:35.875142 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.875082 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rsfpd" Jul 21 00:27:35.881822 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.881796 2580 scope.go:117] "RemoveContainer" containerID="607311a6707cd06b252675349c46431a4f48f9ed138f3ce5d18a9533fd9d3fc8" Jul 21 00:27:35.897449 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.897403 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whtsp" podStartSLOduration=1.897386467 podStartE2EDuration="1.897386467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:27:35.895418352 +0000 UTC m=+650.110798676" watchObservedRunningTime="2026-07-21 00:27:35.897386467 +0000 UTC m=+650.112766791" Jul 21 00:27:35.897732 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:27:35.897706 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8s4tz\" is forbidden: User \"system:node:ip-10-0-139-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-117.ec2.internal' and this object" podUID="1024828c-610d-47eb-9857-c9d99ffdd05b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz" Jul 21 00:27:35.899848 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:27:35.899822 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8s4tz\" is forbidden: User \"system:node:ip-10-0-139-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-117.ec2.internal' and this object" podUID="1024828c-610d-47eb-9857-c9d99ffdd05b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz" Jul 21 00:27:35.901802 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:27:35.901779 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k5rkb\" is forbidden: User \"system:node:ip-10-0-139-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-117.ec2.internal' and this object" podUID="704c12fe-ac99-4d4e-84e3-091159c46428" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" Jul 21 00:27:35.921150 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:35.921088 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rsfpd" podStartSLOduration=1.921067341 podStartE2EDuration="1.921067341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:27:35.919623709 +0000 UTC m=+650.135004026" watchObservedRunningTime="2026-07-21 00:27:35.921067341 +0000 UTC m=+650.136447665" Jul 21 00:27:36.296150 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:27:36.296113 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-8s4tz\" is forbidden: User \"system:node:ip-10-0-139-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-117.ec2.internal' and this object" podUID="1024828c-610d-47eb-9857-c9d99ffdd05b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-8s4tz" Jul 21 00:27:36.298109 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:36.298069 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1024828c-610d-47eb-9857-c9d99ffdd05b" path="/var/lib/kubelet/pods/1024828c-610d-47eb-9857-c9d99ffdd05b/volumes" Jul 21 00:27:36.298213 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:27:36.298121 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k5rkb\" is forbidden: User \"system:node:ip-10-0-139-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-117.ec2.internal' and this object" podUID="704c12fe-ac99-4d4e-84e3-091159c46428" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k5rkb" Jul 21 00:27:36.298432 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:36.298420 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="704c12fe-ac99-4d4e-84e3-091159c46428" path="/var/lib/kubelet/pods/704c12fe-ac99-4d4e-84e3-091159c46428/volumes" Jul 21 00:27:39.900267 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:39.900227 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:40.003440 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:40.003404 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:43.716466 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:43.716432 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:43.783184 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:43.783146 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:46.884842 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:46.884810 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whtsp" Jul 21 00:27:46.885247 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:46.884882 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rsfpd" Jul 21 00:27:52.406081 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:52.406041 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:52.503216 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:52.503178 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:54.847279 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:54.847246 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:54.935244 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:54.935205 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:59.600185 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:59.600144 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:27:59.701298 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:27:59.701251 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:03.622548 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:03.622512 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:03.691183 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:03.691141 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:04.183404 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:04.183362 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:04.283027 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:04.282986 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:07.590356 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:07.590318 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whtsp"] Jul 21 00:28:07.590819 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:07.590534 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whtsp" podUID="8564d093-f96c-49e5-a972-7979677085f3" containerName="manager" containerID="cri-o://70f5590a55c83afe9c89399462f5dea8f6e47b2b288ac625044af8caf8505276" gracePeriod=10 Jul 21 00:28:07.833974 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:07.833946 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whtsp" Jul 21 00:28:07.985828 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:07.985732 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9x5m\" (UniqueName: \"kubernetes.io/projected/8564d093-f96c-49e5-a972-7979677085f3-kube-api-access-q9x5m\") pod \"8564d093-f96c-49e5-a972-7979677085f3\" (UID: \"8564d093-f96c-49e5-a972-7979677085f3\") " Jul 21 00:28:07.985828 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:07.985778 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8564d093-f96c-49e5-a972-7979677085f3-extensions-socket-volume\") pod \"8564d093-f96c-49e5-a972-7979677085f3\" (UID: \"8564d093-f96c-49e5-a972-7979677085f3\") " Jul 21 00:28:07.986196 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:07.986169 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8564d093-f96c-49e5-a972-7979677085f3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8564d093-f96c-49e5-a972-7979677085f3" (UID: "8564d093-f96c-49e5-a972-7979677085f3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:28:07.988036 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:07.988012 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8564d093-f96c-49e5-a972-7979677085f3-kube-api-access-q9x5m" (OuterVolumeSpecName: "kube-api-access-q9x5m") pod "8564d093-f96c-49e5-a972-7979677085f3" (UID: "8564d093-f96c-49e5-a972-7979677085f3"). InnerVolumeSpecName "kube-api-access-q9x5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:28:08.051874 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:08.051837 2580 generic.go:356] "Generic (PLEG): container finished" podID="8564d093-f96c-49e5-a972-7979677085f3" containerID="70f5590a55c83afe9c89399462f5dea8f6e47b2b288ac625044af8caf8505276" exitCode=0 Jul 21 00:28:08.052077 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:08.051896 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whtsp" Jul 21 00:28:08.052077 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:08.051902 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whtsp" event={"ID":"8564d093-f96c-49e5-a972-7979677085f3","Type":"ContainerDied","Data":"70f5590a55c83afe9c89399462f5dea8f6e47b2b288ac625044af8caf8505276"} Jul 21 00:28:08.052077 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:08.051983 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whtsp" event={"ID":"8564d093-f96c-49e5-a972-7979677085f3","Type":"ContainerDied","Data":"ff8e8d86ccd3db1721bdf35323c9d85ecfe615893d8471e881fc2835b1157112"} Jul 21 00:28:08.052077 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:08.052014 2580 scope.go:117] "RemoveContainer" containerID="70f5590a55c83afe9c89399462f5dea8f6e47b2b288ac625044af8caf8505276" Jul 21 00:28:08.061594 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:08.061573 2580 scope.go:117] "RemoveContainer" containerID="70f5590a55c83afe9c89399462f5dea8f6e47b2b288ac625044af8caf8505276" Jul 21 00:28:08.061893 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:28:08.061868 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f5590a55c83afe9c89399462f5dea8f6e47b2b288ac625044af8caf8505276\": container with ID starting with 70f5590a55c83afe9c89399462f5dea8f6e47b2b288ac625044af8caf8505276 not found: ID does not exist" containerID="70f5590a55c83afe9c89399462f5dea8f6e47b2b288ac625044af8caf8505276" Jul 21 00:28:08.061995 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:08.061900 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f5590a55c83afe9c89399462f5dea8f6e47b2b288ac625044af8caf8505276"} err="failed to get container status \"70f5590a55c83afe9c89399462f5dea8f6e47b2b288ac625044af8caf8505276\": rpc error: code = NotFound desc = could not find container \"70f5590a55c83afe9c89399462f5dea8f6e47b2b288ac625044af8caf8505276\": container with ID starting with 70f5590a55c83afe9c89399462f5dea8f6e47b2b288ac625044af8caf8505276 not found: ID does not exist" Jul 21 00:28:08.077362 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:08.077331 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whtsp"] Jul 21 00:28:08.082354 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:08.082327 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-whtsp"] Jul 21 00:28:08.086524 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:08.086503 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9x5m\" (UniqueName: \"kubernetes.io/projected/8564d093-f96c-49e5-a972-7979677085f3-kube-api-access-q9x5m\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:28:08.086592 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:08.086526 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8564d093-f96c-49e5-a972-7979677085f3-extensions-socket-volume\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:28:08.297368 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:08.297284 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8564d093-f96c-49e5-a972-7979677085f3" path="/var/lib/kubelet/pods/8564d093-f96c-49e5-a972-7979677085f3/volumes" Jul 21 00:28:12.811481 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:12.811441 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:12.911364 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:12.911315 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:15.262801 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:15.262743 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:15.318760 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:15.318719 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:20.013206 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:20.013166 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:20.115652 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:20.115587 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:22.173712 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:22.173660 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:22.274388 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:22.274354 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:24.085586 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.085551 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d"] Jul 21 00:28:24.085982 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.085922 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8564d093-f96c-49e5-a972-7979677085f3" containerName="manager" Jul 21 00:28:24.085982 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.085933 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8564d093-f96c-49e5-a972-7979677085f3" containerName="manager" Jul 21 00:28:24.086059 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.086000 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8564d093-f96c-49e5-a972-7979677085f3" containerName="manager" Jul 21 00:28:24.089108 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.089091 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.092696 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.092665 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-729fv\"" Jul 21 00:28:24.112143 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.112113 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d"] Jul 21 00:28:24.116232 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.116200 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944990f8-eece-4bbe-a092-1db9b9547c2f-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.116408 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.116238 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944990f8-eece-4bbe-a092-1db9b9547c2f-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.116408 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.116261 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944990f8-eece-4bbe-a092-1db9b9547c2f-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.116408 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.116313 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxs4q\" (UniqueName: \"kubernetes.io/projected/944990f8-eece-4bbe-a092-1db9b9547c2f-kube-api-access-rxs4q\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.116408 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.116340 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944990f8-eece-4bbe-a092-1db9b9547c2f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.116408 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.116388 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944990f8-eece-4bbe-a092-1db9b9547c2f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.116631 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.116420 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/944990f8-eece-4bbe-a092-1db9b9547c2f-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.116631 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.116440 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944990f8-eece-4bbe-a092-1db9b9547c2f-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.116631 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.116489 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944990f8-eece-4bbe-a092-1db9b9547c2f-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.157187 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.157145 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:24.217756 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.217710 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944990f8-eece-4bbe-a092-1db9b9547c2f-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.217756 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.217755 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944990f8-eece-4bbe-a092-1db9b9547c2f-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.218016 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.217780 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944990f8-eece-4bbe-a092-1db9b9547c2f-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.218016 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.217807 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxs4q\" (UniqueName: \"kubernetes.io/projected/944990f8-eece-4bbe-a092-1db9b9547c2f-kube-api-access-rxs4q\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.218016 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.217837 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944990f8-eece-4bbe-a092-1db9b9547c2f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.218016 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.217861 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944990f8-eece-4bbe-a092-1db9b9547c2f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.218016 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.217902 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/944990f8-eece-4bbe-a092-1db9b9547c2f-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.218270 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.218239 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944990f8-eece-4bbe-a092-1db9b9547c2f-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.218270 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.218253 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944990f8-eece-4bbe-a092-1db9b9547c2f-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.218372 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.218319 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944990f8-eece-4bbe-a092-1db9b9547c2f-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.218372 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.218321 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944990f8-eece-4bbe-a092-1db9b9547c2f-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.218533 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.218506 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944990f8-eece-4bbe-a092-1db9b9547c2f-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.218591 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.218566 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944990f8-eece-4bbe-a092-1db9b9547c2f-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.218661 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.218626 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944990f8-eece-4bbe-a092-1db9b9547c2f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.220773 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.220752 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944990f8-eece-4bbe-a092-1db9b9547c2f-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.221079 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.221056 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944990f8-eece-4bbe-a092-1db9b9547c2f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.260215 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.260178 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/944990f8-eece-4bbe-a092-1db9b9547c2f-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.260396 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.260338 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxs4q\" (UniqueName: \"kubernetes.io/projected/944990f8-eece-4bbe-a092-1db9b9547c2f-kube-api-access-rxs4q\") pod \"maas-default-gateway-openshift-default-56654dcf7-6vs2d\" (UID: \"944990f8-eece-4bbe-a092-1db9b9547c2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.355861 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.355765 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:24.399776 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.399744 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:24.548034 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.548007 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d"] Jul 21 00:28:24.549698 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:28:24.549675 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944990f8_eece_4bbe_a092_1db9b9547c2f.slice/crio-698735e368957f9d48e9b2f9dec99e0619ba649519efdad044b8e40bcf2f8dd9 WatchSource:0}: Error finding container 698735e368957f9d48e9b2f9dec99e0619ba649519efdad044b8e40bcf2f8dd9: Status 404 returned error can't find the container with id 698735e368957f9d48e9b2f9dec99e0619ba649519efdad044b8e40bcf2f8dd9 Jul 21 00:28:24.551778 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.551742 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:28:24.551890 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.551829 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:28:24.551890 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:24.551873 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:28:25.141852 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:25.141820 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" event={"ID":"944990f8-eece-4bbe-a092-1db9b9547c2f","Type":"ContainerStarted","Data":"502f7e92bedd59183f7ca08000fe4a46b21a8c13e6b5b342bba2265a198c9f72"} Jul 21 00:28:25.142255 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:25.141875 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" event={"ID":"944990f8-eece-4bbe-a092-1db9b9547c2f","Type":"ContainerStarted","Data":"698735e368957f9d48e9b2f9dec99e0619ba649519efdad044b8e40bcf2f8dd9"} Jul 21 00:28:25.166781 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:25.166729 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" podStartSLOduration=1.166710343 podStartE2EDuration="1.166710343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:28:25.164117233 +0000 UTC m=+699.379497560" watchObservedRunningTime="2026-07-21 00:28:25.166710343 +0000 UTC m=+699.382090667" Jul 21 00:28:25.400802 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:25.400699 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:25.406109 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:25.406081 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:26.147954 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:26.147917 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:26.149025 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:26.149006 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:28:26.242552 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:26.242511 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:26.243807 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:26.243786 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7mmsq"] Jul 21 00:28:26.246466 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:26.246449 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-7mmsq" Jul 21 00:28:26.249144 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:26.249117 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 21 00:28:26.249144 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:26.249134 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xfbv5\"" Jul 21 00:28:26.254917 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:26.254894 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7mmsq"] Jul 21 00:28:26.335649 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:26.335576 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29fbw\" (UniqueName: \"kubernetes.io/projected/031bd018-4d72-4f9b-b577-319ac73715b7-kube-api-access-29fbw\") pod \"limitador-limitador-64c94fd64f-7mmsq\" (UID: \"031bd018-4d72-4f9b-b577-319ac73715b7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7mmsq" Jul 21 00:28:26.335649 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:26.335634 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/031bd018-4d72-4f9b-b577-319ac73715b7-config-file\") pod \"limitador-limitador-64c94fd64f-7mmsq\" (UID: \"031bd018-4d72-4f9b-b577-319ac73715b7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7mmsq" Jul 21 00:28:26.339400 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:26.339365 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7mmsq"] Jul 21 00:28:26.351412 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:26.351373 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:26.436953 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:26.436837 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29fbw\" (UniqueName: \"kubernetes.io/projected/031bd018-4d72-4f9b-b577-319ac73715b7-kube-api-access-29fbw\") pod \"limitador-limitador-64c94fd64f-7mmsq\" (UID: \"031bd018-4d72-4f9b-b577-319ac73715b7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7mmsq" Jul 21 00:28:26.436953 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:26.436877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/031bd018-4d72-4f9b-b577-319ac73715b7-config-file\") pod \"limitador-limitador-64c94fd64f-7mmsq\" (UID: \"031bd018-4d72-4f9b-b577-319ac73715b7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7mmsq" Jul 21 00:28:26.437723 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:26.437696 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/031bd018-4d72-4f9b-b577-319ac73715b7-config-file\") pod \"limitador-limitador-64c94fd64f-7mmsq\" (UID: \"031bd018-4d72-4f9b-b577-319ac73715b7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7mmsq" Jul 21 00:28:26.446426 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:26.446395 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29fbw\" (UniqueName: \"kubernetes.io/projected/031bd018-4d72-4f9b-b577-319ac73715b7-kube-api-access-29fbw\") pod \"limitador-limitador-64c94fd64f-7mmsq\" (UID: \"031bd018-4d72-4f9b-b577-319ac73715b7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7mmsq" Jul 21 00:28:26.451159 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:26.451121 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:26.559379 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:26.559346 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-7mmsq" Jul 21 00:28:26.702942 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:26.702835 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7mmsq"] Jul 21 00:28:26.705706 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:28:26.705679 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031bd018_4d72_4f9b_b577_319ac73715b7.slice/crio-372a04cd10aa2cb9f8176bee13862f88e053313c98791612d848f16d852765b3 WatchSource:0}: Error finding container 372a04cd10aa2cb9f8176bee13862f88e053313c98791612d848f16d852765b3: Status 404 returned error can't find the container with id 372a04cd10aa2cb9f8176bee13862f88e053313c98791612d848f16d852765b3 Jul 21 00:28:26.707721 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:26.707704 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 00:28:27.028868 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:27.028773 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-8sj8f"] Jul 21 00:28:27.032014 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:27.031994 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8sj8f" Jul 21 00:28:27.034738 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:27.034711 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zx4g5\"" Jul 21 00:28:27.040010 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:27.039793 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-8sj8f"] Jul 21 00:28:27.041957 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:27.041925 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbz7t\" (UniqueName: \"kubernetes.io/projected/8724ef6f-e170-4bc2-ac96-9050bbcd4d5b-kube-api-access-xbz7t\") pod \"authorino-58cd955675-8sj8f\" (UID: \"8724ef6f-e170-4bc2-ac96-9050bbcd4d5b\") " pod="kuadrant-system/authorino-58cd955675-8sj8f" Jul 21 00:28:27.065358 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:27.065327 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:27.142447 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:27.142411 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbz7t\" (UniqueName: \"kubernetes.io/projected/8724ef6f-e170-4bc2-ac96-9050bbcd4d5b-kube-api-access-xbz7t\") pod \"authorino-58cd955675-8sj8f\" (UID: \"8724ef6f-e170-4bc2-ac96-9050bbcd4d5b\") " pod="kuadrant-system/authorino-58cd955675-8sj8f" Jul 21 00:28:27.142869 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:27.142850 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:27.154557 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:27.154527 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbz7t\" (UniqueName: \"kubernetes.io/projected/8724ef6f-e170-4bc2-ac96-9050bbcd4d5b-kube-api-access-xbz7t\") pod \"authorino-58cd955675-8sj8f\" (UID: \"8724ef6f-e170-4bc2-ac96-9050bbcd4d5b\") " pod="kuadrant-system/authorino-58cd955675-8sj8f" Jul 21 00:28:27.155051 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:27.154588 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-7mmsq" event={"ID":"031bd018-4d72-4f9b-b577-319ac73715b7","Type":"ContainerStarted","Data":"372a04cd10aa2cb9f8176bee13862f88e053313c98791612d848f16d852765b3"} Jul 21 00:28:27.342917 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:27.342884 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8sj8f" Jul 21 00:28:27.485108 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:27.485077 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-8sj8f"] Jul 21 00:28:27.485893 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:28:27.485872 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8724ef6f_e170_4bc2_ac96_9050bbcd4d5b.slice/crio-44709b392f513a19304bbdbcb11c822ece4637e09a6b749942b66b9e7db85116 WatchSource:0}: Error finding container 44709b392f513a19304bbdbcb11c822ece4637e09a6b749942b66b9e7db85116: Status 404 returned error can't find the container with id 44709b392f513a19304bbdbcb11c822ece4637e09a6b749942b66b9e7db85116 Jul 21 00:28:28.163013 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:28.162969 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8sj8f" event={"ID":"8724ef6f-e170-4bc2-ac96-9050bbcd4d5b","Type":"ContainerStarted","Data":"44709b392f513a19304bbdbcb11c822ece4637e09a6b749942b66b9e7db85116"} Jul 21 00:28:30.180256 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:30.180208 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-7mmsq" event={"ID":"031bd018-4d72-4f9b-b577-319ac73715b7","Type":"ContainerStarted","Data":"57b25c8a710b1361715fadce7ce32ad6e422f30c70f37e22ce768eb66c1f9d99"} Jul 21 00:28:30.180734 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:30.180407 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-7mmsq" Jul 21 00:28:30.199143 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:30.199072 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-7mmsq" podStartSLOduration=1.231897233 podStartE2EDuration="4.199048502s" totalImagesPullingTime="2.967151269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:28:30.196451068 +0000 UTC m=+704.411831394" watchObservedRunningTime="2026-07-21 00:28:30.199048502 +0000 UTC m=+704.414428828" Jul 21 00:28:31.737302 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:31.737214 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-8sj8f"] Jul 21 00:28:32.193235 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:32.193188 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8sj8f" event={"ID":"8724ef6f-e170-4bc2-ac96-9050bbcd4d5b","Type":"ContainerStarted","Data":"ab9caa2ec4ff90b0e3723f13adb92ba3732208d87411e01229b8cb0d20087733"} Jul 21 00:28:32.208794 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:32.208739 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-8sj8f" podStartSLOduration=1.215327903 podStartE2EDuration="5.208721935s" totalImagesPullingTime="3.993394032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:28:32.206329982 +0000 UTC m=+706.421710306" watchObservedRunningTime="2026-07-21 00:28:32.208721935 +0000 UTC m=+706.424102259" Jul 21 00:28:33.198061 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:33.198021 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-8sj8f" podUID="8724ef6f-e170-4bc2-ac96-9050bbcd4d5b" containerName="authorino" containerID="cri-o://ab9caa2ec4ff90b0e3723f13adb92ba3732208d87411e01229b8cb0d20087733" gracePeriod=30 Jul 21 00:28:33.450151 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:33.450081 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8sj8f" Jul 21 00:28:33.501666 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:33.501634 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbz7t\" (UniqueName: \"kubernetes.io/projected/8724ef6f-e170-4bc2-ac96-9050bbcd4d5b-kube-api-access-xbz7t\") pod \"8724ef6f-e170-4bc2-ac96-9050bbcd4d5b\" (UID: \"8724ef6f-e170-4bc2-ac96-9050bbcd4d5b\") " Jul 21 00:28:33.503987 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:33.503947 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8724ef6f-e170-4bc2-ac96-9050bbcd4d5b-kube-api-access-xbz7t" (OuterVolumeSpecName: "kube-api-access-xbz7t") pod "8724ef6f-e170-4bc2-ac96-9050bbcd4d5b" (UID: "8724ef6f-e170-4bc2-ac96-9050bbcd4d5b"). InnerVolumeSpecName "kube-api-access-xbz7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:28:33.602715 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:33.602675 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbz7t\" (UniqueName: \"kubernetes.io/projected/8724ef6f-e170-4bc2-ac96-9050bbcd4d5b-kube-api-access-xbz7t\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:28:34.205071 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:34.205036 2580 generic.go:356] "Generic (PLEG): container finished" podID="8724ef6f-e170-4bc2-ac96-9050bbcd4d5b" containerID="ab9caa2ec4ff90b0e3723f13adb92ba3732208d87411e01229b8cb0d20087733" exitCode=0 Jul 21 00:28:34.205071 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:34.205075 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8sj8f" Jul 21 00:28:34.205572 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:34.205117 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8sj8f" event={"ID":"8724ef6f-e170-4bc2-ac96-9050bbcd4d5b","Type":"ContainerDied","Data":"ab9caa2ec4ff90b0e3723f13adb92ba3732208d87411e01229b8cb0d20087733"} Jul 21 00:28:34.205572 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:34.205160 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8sj8f" event={"ID":"8724ef6f-e170-4bc2-ac96-9050bbcd4d5b","Type":"ContainerDied","Data":"44709b392f513a19304bbdbcb11c822ece4637e09a6b749942b66b9e7db85116"} Jul 21 00:28:34.205572 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:34.205177 2580 scope.go:117] "RemoveContainer" containerID="ab9caa2ec4ff90b0e3723f13adb92ba3732208d87411e01229b8cb0d20087733" Jul 21 00:28:34.221397 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:34.221375 2580 scope.go:117] "RemoveContainer" containerID="ab9caa2ec4ff90b0e3723f13adb92ba3732208d87411e01229b8cb0d20087733" Jul 21 00:28:34.221760 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:28:34.221740 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab9caa2ec4ff90b0e3723f13adb92ba3732208d87411e01229b8cb0d20087733\": container with ID starting with ab9caa2ec4ff90b0e3723f13adb92ba3732208d87411e01229b8cb0d20087733 not found: ID does not exist" containerID="ab9caa2ec4ff90b0e3723f13adb92ba3732208d87411e01229b8cb0d20087733" Jul 21 00:28:34.221817 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:34.221770 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9caa2ec4ff90b0e3723f13adb92ba3732208d87411e01229b8cb0d20087733"} err="failed to get container status \"ab9caa2ec4ff90b0e3723f13adb92ba3732208d87411e01229b8cb0d20087733\": rpc error: code = NotFound desc = could not find container \"ab9caa2ec4ff90b0e3723f13adb92ba3732208d87411e01229b8cb0d20087733\": container with ID starting with ab9caa2ec4ff90b0e3723f13adb92ba3732208d87411e01229b8cb0d20087733 not found: ID does not exist" Jul 21 00:28:34.230375 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:34.230350 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-8sj8f"] Jul 21 00:28:34.233849 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:34.233818 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-8sj8f"] Jul 21 00:28:34.297463 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:34.297429 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8724ef6f-e170-4bc2-ac96-9050bbcd4d5b" path="/var/lib/kubelet/pods/8724ef6f-e170-4bc2-ac96-9050bbcd4d5b/volumes" Jul 21 00:28:40.796963 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:40.796881 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7mmsq"] Jul 21 00:28:40.797475 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:40.797100 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-7mmsq" podUID="031bd018-4d72-4f9b-b577-319ac73715b7" containerName="limitador" containerID="cri-o://57b25c8a710b1361715fadce7ce32ad6e422f30c70f37e22ce768eb66c1f9d99" gracePeriod=30 Jul 21 00:28:40.797874 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:40.797847 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-7mmsq" Jul 21 00:28:41.246193 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:41.246163 2580 generic.go:356] "Generic (PLEG): container finished" podID="031bd018-4d72-4f9b-b577-319ac73715b7" containerID="57b25c8a710b1361715fadce7ce32ad6e422f30c70f37e22ce768eb66c1f9d99" exitCode=0 Jul 21 00:28:41.246394 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:41.246232 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-7mmsq" event={"ID":"031bd018-4d72-4f9b-b577-319ac73715b7","Type":"ContainerDied","Data":"57b25c8a710b1361715fadce7ce32ad6e422f30c70f37e22ce768eb66c1f9d99"} Jul 21 00:28:41.357163 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:41.357094 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-7mmsq" Jul 21 00:28:41.469923 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:41.469888 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/031bd018-4d72-4f9b-b577-319ac73715b7-config-file\") pod \"031bd018-4d72-4f9b-b577-319ac73715b7\" (UID: \"031bd018-4d72-4f9b-b577-319ac73715b7\") " Jul 21 00:28:41.469923 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:41.469928 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29fbw\" (UniqueName: \"kubernetes.io/projected/031bd018-4d72-4f9b-b577-319ac73715b7-kube-api-access-29fbw\") pod \"031bd018-4d72-4f9b-b577-319ac73715b7\" (UID: \"031bd018-4d72-4f9b-b577-319ac73715b7\") " Jul 21 00:28:41.470353 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:41.470325 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/031bd018-4d72-4f9b-b577-319ac73715b7-config-file" (OuterVolumeSpecName: "config-file") pod "031bd018-4d72-4f9b-b577-319ac73715b7" (UID: "031bd018-4d72-4f9b-b577-319ac73715b7"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:28:41.472144 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:41.472123 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031bd018-4d72-4f9b-b577-319ac73715b7-kube-api-access-29fbw" (OuterVolumeSpecName: "kube-api-access-29fbw") pod "031bd018-4d72-4f9b-b577-319ac73715b7" (UID: "031bd018-4d72-4f9b-b577-319ac73715b7"). InnerVolumeSpecName "kube-api-access-29fbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:28:41.571372 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:41.571333 2580 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/031bd018-4d72-4f9b-b577-319ac73715b7-config-file\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:28:41.571372 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:41.571363 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29fbw\" (UniqueName: \"kubernetes.io/projected/031bd018-4d72-4f9b-b577-319ac73715b7-kube-api-access-29fbw\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:28:42.252735 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:42.252706 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-7mmsq" Jul 21 00:28:42.252735 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:42.252733 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-7mmsq" event={"ID":"031bd018-4d72-4f9b-b577-319ac73715b7","Type":"ContainerDied","Data":"372a04cd10aa2cb9f8176bee13862f88e053313c98791612d848f16d852765b3"} Jul 21 00:28:42.253263 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:42.252789 2580 scope.go:117] "RemoveContainer" containerID="57b25c8a710b1361715fadce7ce32ad6e422f30c70f37e22ce768eb66c1f9d99" Jul 21 00:28:42.274584 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:42.274557 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7mmsq"] Jul 21 00:28:42.278198 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:42.278172 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7mmsq"] Jul 21 00:28:42.297377 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:42.297346 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="031bd018-4d72-4f9b-b577-319ac73715b7" path="/var/lib/kubelet/pods/031bd018-4d72-4f9b-b577-319ac73715b7/volumes" Jul 21 00:28:42.465546 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:42.465511 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:42.564416 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:42.564391 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:42.910702 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:42.910615 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-dw8m6"] Jul 21 00:28:42.910977 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:42.910965 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8724ef6f-e170-4bc2-ac96-9050bbcd4d5b" containerName="authorino" Jul 21 00:28:42.911041 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:42.910978 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8724ef6f-e170-4bc2-ac96-9050bbcd4d5b" containerName="authorino" Jul 21 00:28:42.911041 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:42.910992 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="031bd018-4d72-4f9b-b577-319ac73715b7" containerName="limitador" Jul 21 00:28:42.911041 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:42.910998 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="031bd018-4d72-4f9b-b577-319ac73715b7" containerName="limitador" Jul 21 00:28:42.911189 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:42.911050 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="031bd018-4d72-4f9b-b577-319ac73715b7" containerName="limitador" Jul 21 00:28:42.911189 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:42.911058 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8724ef6f-e170-4bc2-ac96-9050bbcd4d5b" containerName="authorino" Jul 21 00:28:42.915446 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:42.915423 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dw8m6" Jul 21 00:28:42.918074 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:42.918047 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 00:28:42.918342 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:42.918320 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-tcdvf\"" Jul 21 00:28:42.918451 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:42.918366 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 21 00:28:42.919315 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:42.919294 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 21 00:28:42.923301 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:42.923272 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-dw8m6"] Jul 21 00:28:42.940750 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:42.940706 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:42.986248 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:42.986212 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/12b4b9f5-5148-400d-834a-f1822f8eacfb-data\") pod \"postgres-69fb4594cf-dw8m6\" (UID: \"12b4b9f5-5148-400d-834a-f1822f8eacfb\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-dw8m6" Jul 21 00:28:42.986419 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:42.986256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9cn5\" (UniqueName: \"kubernetes.io/projected/12b4b9f5-5148-400d-834a-f1822f8eacfb-kube-api-access-j9cn5\") pod \"postgres-69fb4594cf-dw8m6\" (UID: \"12b4b9f5-5148-400d-834a-f1822f8eacfb\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-dw8m6" Jul 21 00:28:43.087379 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:43.087336 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9cn5\" (UniqueName: \"kubernetes.io/projected/12b4b9f5-5148-400d-834a-f1822f8eacfb-kube-api-access-j9cn5\") pod \"postgres-69fb4594cf-dw8m6\" (UID: \"12b4b9f5-5148-400d-834a-f1822f8eacfb\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-dw8m6" Jul 21 00:28:43.087565 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:43.087468 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/12b4b9f5-5148-400d-834a-f1822f8eacfb-data\") pod \"postgres-69fb4594cf-dw8m6\" (UID: \"12b4b9f5-5148-400d-834a-f1822f8eacfb\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-dw8m6" Jul 21 00:28:43.087891 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:43.087874 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/12b4b9f5-5148-400d-834a-f1822f8eacfb-data\") pod \"postgres-69fb4594cf-dw8m6\" (UID: \"12b4b9f5-5148-400d-834a-f1822f8eacfb\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-dw8m6" Jul 21 00:28:43.096394 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:43.096364 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9cn5\" (UniqueName: \"kubernetes.io/projected/12b4b9f5-5148-400d-834a-f1822f8eacfb-kube-api-access-j9cn5\") pod \"postgres-69fb4594cf-dw8m6\" (UID: \"12b4b9f5-5148-400d-834a-f1822f8eacfb\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-dw8m6" Jul 21 00:28:43.228423 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:43.228321 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dw8m6" Jul 21 00:28:43.365293 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:43.365266 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-dw8m6"] Jul 21 00:28:43.366879 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:28:43.366853 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b4b9f5_5148_400d_834a_f1822f8eacfb.slice/crio-df9bfb50f1d515a324a9392608a13a1c973a4cac107e9f56ab4a3215cb798d82 WatchSource:0}: Error finding container df9bfb50f1d515a324a9392608a13a1c973a4cac107e9f56ab4a3215cb798d82: Status 404 returned error can't find the container with id df9bfb50f1d515a324a9392608a13a1c973a4cac107e9f56ab4a3215cb798d82 Jul 21 00:28:44.268623 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:44.268564 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dw8m6" event={"ID":"12b4b9f5-5148-400d-834a-f1822f8eacfb","Type":"ContainerStarted","Data":"df9bfb50f1d515a324a9392608a13a1c973a4cac107e9f56ab4a3215cb798d82"} Jul 21 00:28:47.933014 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:47.932993 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 21 00:28:48.298002 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:48.297922 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dw8m6" Jul 21 00:28:48.298002 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:48.297975 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dw8m6" event={"ID":"12b4b9f5-5148-400d-834a-f1822f8eacfb","Type":"ContainerStarted","Data":"6d910057532a791d4a9f28436349232c8c4819365f3ef565dac45e63dbc596d6"} Jul 21 00:28:48.312947 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:48.312898 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dw8m6" podStartSLOduration=1.7507798380000001 podStartE2EDuration="6.31288046s" totalImagesPullingTime="4.562100622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:28:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:28:48.310814492 +0000 UTC m=+722.526194819" watchObservedRunningTime="2026-07-21 00:28:48.31288046 +0000 UTC m=+722.528260819" Jul 21 00:28:54.330717 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:54.330686 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-dw8m6" Jul 21 00:28:54.661162 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:54.661051 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:54.761737 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:54.761699 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:28:54.921505 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:54.921426 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7cpxp"] Jul 21 00:28:54.927664 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:54.927635 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7cpxp" Jul 21 00:28:54.930433 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:54.930405 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zx4g5\"" Jul 21 00:28:54.932573 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:54.932547 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7cpxp"] Jul 21 00:28:54.995953 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:54.995909 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfvlg\" (UniqueName: \"kubernetes.io/projected/6bc886b4-0462-414e-9d3d-e6c25eea8017-kube-api-access-kfvlg\") pod \"authorino-5d4b67c446-7cpxp\" (UID: \"6bc886b4-0462-414e-9d3d-e6c25eea8017\") " pod="kuadrant-system/authorino-5d4b67c446-7cpxp" Jul 21 00:28:55.088372 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.088331 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7cpxp"] Jul 21 00:28:55.088643 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:28:55.088623 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-kfvlg], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-7cpxp" podUID="6bc886b4-0462-414e-9d3d-e6c25eea8017" Jul 21 00:28:55.097297 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.097266 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfvlg\" (UniqueName: \"kubernetes.io/projected/6bc886b4-0462-414e-9d3d-e6c25eea8017-kube-api-access-kfvlg\") pod \"authorino-5d4b67c446-7cpxp\" (UID: \"6bc886b4-0462-414e-9d3d-e6c25eea8017\") " pod="kuadrant-system/authorino-5d4b67c446-7cpxp" Jul 21 00:28:55.110943 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.110912 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfvlg\" (UniqueName: \"kubernetes.io/projected/6bc886b4-0462-414e-9d3d-e6c25eea8017-kube-api-access-kfvlg\") pod \"authorino-5d4b67c446-7cpxp\" (UID: \"6bc886b4-0462-414e-9d3d-e6c25eea8017\") " pod="kuadrant-system/authorino-5d4b67c446-7cpxp" Jul 21 00:28:55.116098 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.116067 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-qkzfg"] Jul 21 00:28:55.120520 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.120487 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-qkzfg" Jul 21 00:28:55.123235 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.123214 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 21 00:28:55.129461 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.129435 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-qkzfg"] Jul 21 00:28:55.184004 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.183915 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-qkzfg"] Jul 21 00:28:55.184208 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:28:55.184189 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-nrpqq tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-76877994-qkzfg" podUID="8dae4c2c-98f4-4c0d-af7d-950e91096489" Jul 21 00:28:55.198092 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.198059 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrpqq\" (UniqueName: \"kubernetes.io/projected/8dae4c2c-98f4-4c0d-af7d-950e91096489-kube-api-access-nrpqq\") pod \"authorino-76877994-qkzfg\" (UID: \"8dae4c2c-98f4-4c0d-af7d-950e91096489\") " pod="kuadrant-system/authorino-76877994-qkzfg" Jul 21 00:28:55.198185 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.198138 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8dae4c2c-98f4-4c0d-af7d-950e91096489-tls-cert\") pod \"authorino-76877994-qkzfg\" (UID: \"8dae4c2c-98f4-4c0d-af7d-950e91096489\") " pod="kuadrant-system/authorino-76877994-qkzfg" Jul 21 00:28:55.212202 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.212166 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7f755bfcdc-qqx9g"] Jul 21 00:28:55.216274 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.216253 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f755bfcdc-qqx9g" Jul 21 00:28:55.231567 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.231527 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f755bfcdc-qqx9g"] Jul 21 00:28:55.299281 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.299240 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brqbz\" (UniqueName: \"kubernetes.io/projected/20161088-5d29-4f83-b8d8-0a4292c9d959-kube-api-access-brqbz\") pod \"authorino-7f755bfcdc-qqx9g\" (UID: \"20161088-5d29-4f83-b8d8-0a4292c9d959\") " pod="kuadrant-system/authorino-7f755bfcdc-qqx9g" Jul 21 00:28:55.299281 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.299287 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/20161088-5d29-4f83-b8d8-0a4292c9d959-tls-cert\") pod \"authorino-7f755bfcdc-qqx9g\" (UID: \"20161088-5d29-4f83-b8d8-0a4292c9d959\") " pod="kuadrant-system/authorino-7f755bfcdc-qqx9g" Jul 21 00:28:55.299567 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.299353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8dae4c2c-98f4-4c0d-af7d-950e91096489-tls-cert\") pod \"authorino-76877994-qkzfg\" (UID: \"8dae4c2c-98f4-4c0d-af7d-950e91096489\") " pod="kuadrant-system/authorino-76877994-qkzfg" Jul 21 00:28:55.299567 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.299415 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrpqq\" (UniqueName: \"kubernetes.io/projected/8dae4c2c-98f4-4c0d-af7d-950e91096489-kube-api-access-nrpqq\") pod \"authorino-76877994-qkzfg\" (UID: \"8dae4c2c-98f4-4c0d-af7d-950e91096489\") " pod="kuadrant-system/authorino-76877994-qkzfg" Jul 21 00:28:55.302047 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.302019 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8dae4c2c-98f4-4c0d-af7d-950e91096489-tls-cert\") pod \"authorino-76877994-qkzfg\" (UID: \"8dae4c2c-98f4-4c0d-af7d-950e91096489\") " pod="kuadrant-system/authorino-76877994-qkzfg" Jul 21 00:28:55.307762 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.307736 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrpqq\" (UniqueName: \"kubernetes.io/projected/8dae4c2c-98f4-4c0d-af7d-950e91096489-kube-api-access-nrpqq\") pod \"authorino-76877994-qkzfg\" (UID: \"8dae4c2c-98f4-4c0d-af7d-950e91096489\") " pod="kuadrant-system/authorino-76877994-qkzfg" Jul 21 00:28:55.331823 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.331791 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-qkzfg" Jul 21 00:28:55.332233 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.331794 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7cpxp" Jul 21 00:28:55.340113 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.340088 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-qkzfg" Jul 21 00:28:55.343290 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.343260 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7cpxp" Jul 21 00:28:55.400430 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.400390 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brqbz\" (UniqueName: \"kubernetes.io/projected/20161088-5d29-4f83-b8d8-0a4292c9d959-kube-api-access-brqbz\") pod \"authorino-7f755bfcdc-qqx9g\" (UID: \"20161088-5d29-4f83-b8d8-0a4292c9d959\") " pod="kuadrant-system/authorino-7f755bfcdc-qqx9g" Jul 21 00:28:55.400671 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.400468 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/20161088-5d29-4f83-b8d8-0a4292c9d959-tls-cert\") pod \"authorino-7f755bfcdc-qqx9g\" (UID: \"20161088-5d29-4f83-b8d8-0a4292c9d959\") " pod="kuadrant-system/authorino-7f755bfcdc-qqx9g" Jul 21 00:28:55.403091 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.403069 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/20161088-5d29-4f83-b8d8-0a4292c9d959-tls-cert\") pod \"authorino-7f755bfcdc-qqx9g\" (UID: \"20161088-5d29-4f83-b8d8-0a4292c9d959\") " pod="kuadrant-system/authorino-7f755bfcdc-qqx9g" Jul 21 00:28:55.410329 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.410303 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brqbz\" (UniqueName: \"kubernetes.io/projected/20161088-5d29-4f83-b8d8-0a4292c9d959-kube-api-access-brqbz\") pod \"authorino-7f755bfcdc-qqx9g\" (UID: \"20161088-5d29-4f83-b8d8-0a4292c9d959\") " pod="kuadrant-system/authorino-7f755bfcdc-qqx9g" Jul 21 00:28:55.501146 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.501049 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfvlg\" (UniqueName: \"kubernetes.io/projected/6bc886b4-0462-414e-9d3d-e6c25eea8017-kube-api-access-kfvlg\") pod \"6bc886b4-0462-414e-9d3d-e6c25eea8017\" (UID: \"6bc886b4-0462-414e-9d3d-e6c25eea8017\") " Jul 21 00:28:55.501146 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.501109 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrpqq\" (UniqueName: \"kubernetes.io/projected/8dae4c2c-98f4-4c0d-af7d-950e91096489-kube-api-access-nrpqq\") pod \"8dae4c2c-98f4-4c0d-af7d-950e91096489\" (UID: \"8dae4c2c-98f4-4c0d-af7d-950e91096489\") " Jul 21 00:28:55.501146 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.501134 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8dae4c2c-98f4-4c0d-af7d-950e91096489-tls-cert\") pod \"8dae4c2c-98f4-4c0d-af7d-950e91096489\" (UID: \"8dae4c2c-98f4-4c0d-af7d-950e91096489\") " Jul 21 00:28:55.503416 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.503384 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dae4c2c-98f4-4c0d-af7d-950e91096489-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "8dae4c2c-98f4-4c0d-af7d-950e91096489" (UID: "8dae4c2c-98f4-4c0d-af7d-950e91096489"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:28:55.503416 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.503400 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dae4c2c-98f4-4c0d-af7d-950e91096489-kube-api-access-nrpqq" (OuterVolumeSpecName: "kube-api-access-nrpqq") pod "8dae4c2c-98f4-4c0d-af7d-950e91096489" (UID: "8dae4c2c-98f4-4c0d-af7d-950e91096489"). InnerVolumeSpecName "kube-api-access-nrpqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:28:55.503416 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.503387 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc886b4-0462-414e-9d3d-e6c25eea8017-kube-api-access-kfvlg" (OuterVolumeSpecName: "kube-api-access-kfvlg") pod "6bc886b4-0462-414e-9d3d-e6c25eea8017" (UID: "6bc886b4-0462-414e-9d3d-e6c25eea8017"). InnerVolumeSpecName "kube-api-access-kfvlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:28:55.525638 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.525579 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f755bfcdc-qqx9g" Jul 21 00:28:55.602395 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.602365 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfvlg\" (UniqueName: \"kubernetes.io/projected/6bc886b4-0462-414e-9d3d-e6c25eea8017-kube-api-access-kfvlg\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:28:55.602395 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.602392 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrpqq\" (UniqueName: \"kubernetes.io/projected/8dae4c2c-98f4-4c0d-af7d-950e91096489-kube-api-access-nrpqq\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:28:55.602395 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.602403 2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8dae4c2c-98f4-4c0d-af7d-950e91096489-tls-cert\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:28:55.664661 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:55.664637 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f755bfcdc-qqx9g"] Jul 21 00:28:55.666698 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:28:55.666670 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20161088_5d29_4f83_b8d8_0a4292c9d959.slice/crio-ed9d1e58721878bd19f7481f7eeea80798130efe4e34229df6569187c4dc7a3d WatchSource:0}: Error finding container ed9d1e58721878bd19f7481f7eeea80798130efe4e34229df6569187c4dc7a3d: Status 404 returned error can't find the container with id ed9d1e58721878bd19f7481f7eeea80798130efe4e34229df6569187c4dc7a3d Jul 21 00:28:56.338528 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:56.338502 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-qkzfg" Jul 21 00:28:56.338528 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:56.338522 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7cpxp" Jul 21 00:28:56.338528 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:56.338497 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f755bfcdc-qqx9g" event={"ID":"20161088-5d29-4f83-b8d8-0a4292c9d959","Type":"ContainerStarted","Data":"e6e950adf727ed9e706057dd736889d4da3129f4fca7050ecb78c6803f0329ad"} Jul 21 00:28:56.339013 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:56.338566 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f755bfcdc-qqx9g" event={"ID":"20161088-5d29-4f83-b8d8-0a4292c9d959","Type":"ContainerStarted","Data":"ed9d1e58721878bd19f7481f7eeea80798130efe4e34229df6569187c4dc7a3d"} Jul 21 00:28:56.353341 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:56.353239 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7f755bfcdc-qqx9g" podStartSLOduration=0.928582668 podStartE2EDuration="1.353221197s" totalImagesPullingTime="424.638529ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:28:56.352391565 +0000 UTC m=+730.567771890" watchObservedRunningTime="2026-07-21 00:28:56.353221197 +0000 UTC m=+730.568601527" Jul 21 00:28:56.378472 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:56.378437 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-qkzfg"] Jul 21 00:28:56.387915 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:56.387882 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-qkzfg"] Jul 21 00:28:56.409981 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:56.409952 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7cpxp"] Jul 21 00:28:56.413361 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:56.413331 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7cpxp"] Jul 21 00:28:58.297717 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:58.297682 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bc886b4-0462-414e-9d3d-e6c25eea8017" path="/var/lib/kubelet/pods/6bc886b4-0462-414e-9d3d-e6c25eea8017/volumes" Jul 21 00:28:58.298103 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:28:58.297937 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dae4c2c-98f4-4c0d-af7d-950e91096489" path="/var/lib/kubelet/pods/8dae4c2c-98f4-4c0d-af7d-950e91096489/volumes" Jul 21 00:29:01.312243 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:01.312206 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:29:01.325994 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:01.325966 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78467bdbc7-fw55x"] Jul 21 00:29:01.416971 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:01.416933 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:29:01.423692 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:01.423662 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78467bdbc7-fw55x"] Jul 21 00:29:01.423835 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:01.423776 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78467bdbc7-fw55x" Jul 21 00:29:01.428100 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:01.428070 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 00:29:01.428281 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:01.428130 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 00:29:01.428281 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:01.428130 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nhqrc\"" Jul 21 00:29:01.499008 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:01.498974 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:29:01.555481 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:01.555442 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9579e94-e396-4f13-a829-0829edc99f4d-webhook-cert\") pod \"maas-controller-78467bdbc7-fw55x\" (UID: \"d9579e94-e396-4f13-a829-0829edc99f4d\") " pod="opendatahub/maas-controller-78467bdbc7-fw55x" Jul 21 00:29:01.555695 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:01.555511 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkd62\" (UniqueName: \"kubernetes.io/projected/d9579e94-e396-4f13-a829-0829edc99f4d-kube-api-access-xkd62\") pod \"maas-controller-78467bdbc7-fw55x\" (UID: \"d9579e94-e396-4f13-a829-0829edc99f4d\") " pod="opendatahub/maas-controller-78467bdbc7-fw55x" Jul 21 00:29:01.656338 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:01.656245 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9579e94-e396-4f13-a829-0829edc99f4d-webhook-cert\") pod \"maas-controller-78467bdbc7-fw55x\" (UID: \"d9579e94-e396-4f13-a829-0829edc99f4d\") " pod="opendatahub/maas-controller-78467bdbc7-fw55x" Jul 21 00:29:01.656338 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:01.656307 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkd62\" (UniqueName: \"kubernetes.io/projected/d9579e94-e396-4f13-a829-0829edc99f4d-kube-api-access-xkd62\") pod \"maas-controller-78467bdbc7-fw55x\" (UID: \"d9579e94-e396-4f13-a829-0829edc99f4d\") " pod="opendatahub/maas-controller-78467bdbc7-fw55x" Jul 21 00:29:01.658971 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:01.658942 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9579e94-e396-4f13-a829-0829edc99f4d-webhook-cert\") pod \"maas-controller-78467bdbc7-fw55x\" (UID: \"d9579e94-e396-4f13-a829-0829edc99f4d\") " pod="opendatahub/maas-controller-78467bdbc7-fw55x" Jul 21 00:29:01.665929 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:01.665899 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkd62\" (UniqueName: \"kubernetes.io/projected/d9579e94-e396-4f13-a829-0829edc99f4d-kube-api-access-xkd62\") pod \"maas-controller-78467bdbc7-fw55x\" (UID: \"d9579e94-e396-4f13-a829-0829edc99f4d\") " pod="opendatahub/maas-controller-78467bdbc7-fw55x" Jul 21 00:29:01.734071 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:01.734032 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78467bdbc7-fw55x" Jul 21 00:29:01.893523 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:01.893494 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78467bdbc7-fw55x"] Jul 21 00:29:01.895432 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:29:01.895402 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9579e94_e396_4f13_a829_0829edc99f4d.slice/crio-666715bac384b9c48ff7fe7ddc0aeb3139398cd9298323288ddfab09f872e00e WatchSource:0}: Error finding container 666715bac384b9c48ff7fe7ddc0aeb3139398cd9298323288ddfab09f872e00e: Status 404 returned error can't find the container with id 666715bac384b9c48ff7fe7ddc0aeb3139398cd9298323288ddfab09f872e00e Jul 21 00:29:02.375250 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:02.375215 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78467bdbc7-fw55x" event={"ID":"d9579e94-e396-4f13-a829-0829edc99f4d","Type":"ContainerStarted","Data":"666715bac384b9c48ff7fe7ddc0aeb3139398cd9298323288ddfab09f872e00e"} Jul 21 00:29:04.362050 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:04.362000 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:29:04.460171 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:04.460137 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:29:05.397372 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:05.397276 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78467bdbc7-fw55x" event={"ID":"d9579e94-e396-4f13-a829-0829edc99f4d","Type":"ContainerStarted","Data":"e8d52e83bed2f487b3e01d2335ec8e2671179ab5a2e0386f3376814537059904"} Jul 21 00:29:05.397372 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:05.397356 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78467bdbc7-fw55x" Jul 21 00:29:05.421254 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:05.421190 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-78467bdbc7-fw55x" podStartSLOduration=1.224656264 podStartE2EDuration="4.421166792s" totalImagesPullingTime="3.196510528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:29:05.417317297 +0000 UTC m=+739.632697623" watchObservedRunningTime="2026-07-21 00:29:05.421166792 +0000 UTC m=+739.636547121" Jul 21 00:29:15.911352 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:15.911314 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78467bdbc7-fw55x"] Jul 21 00:29:15.911904 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:15.911532 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-78467bdbc7-fw55x" podUID="d9579e94-e396-4f13-a829-0829edc99f4d" containerName="manager" containerID="cri-o://e8d52e83bed2f487b3e01d2335ec8e2671179ab5a2e0386f3376814537059904" gracePeriod=30 Jul 21 00:29:15.913696 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:15.913667 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78467bdbc7-fw55x" Jul 21 00:29:16.160891 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:16.160867 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78467bdbc7-fw55x" Jul 21 00:29:16.183034 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:16.182948 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9579e94-e396-4f13-a829-0829edc99f4d-webhook-cert\") pod \"d9579e94-e396-4f13-a829-0829edc99f4d\" (UID: \"d9579e94-e396-4f13-a829-0829edc99f4d\") " Jul 21 00:29:16.183034 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:16.182997 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkd62\" (UniqueName: \"kubernetes.io/projected/d9579e94-e396-4f13-a829-0829edc99f4d-kube-api-access-xkd62\") pod \"d9579e94-e396-4f13-a829-0829edc99f4d\" (UID: \"d9579e94-e396-4f13-a829-0829edc99f4d\") " Jul 21 00:29:16.185524 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:16.185490 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9579e94-e396-4f13-a829-0829edc99f4d-kube-api-access-xkd62" (OuterVolumeSpecName: "kube-api-access-xkd62") pod "d9579e94-e396-4f13-a829-0829edc99f4d" (UID: "d9579e94-e396-4f13-a829-0829edc99f4d"). InnerVolumeSpecName "kube-api-access-xkd62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:29:16.185524 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:16.185504 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9579e94-e396-4f13-a829-0829edc99f4d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d9579e94-e396-4f13-a829-0829edc99f4d" (UID: "d9579e94-e396-4f13-a829-0829edc99f4d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:29:16.284317 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:16.284283 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9579e94-e396-4f13-a829-0829edc99f4d-webhook-cert\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:29:16.284317 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:16.284313 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkd62\" (UniqueName: \"kubernetes.io/projected/d9579e94-e396-4f13-a829-0829edc99f4d-kube-api-access-xkd62\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:29:16.459201 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:16.459118 2580 generic.go:356] "Generic (PLEG): container finished" podID="d9579e94-e396-4f13-a829-0829edc99f4d" containerID="e8d52e83bed2f487b3e01d2335ec8e2671179ab5a2e0386f3376814537059904" exitCode=0 Jul 21 00:29:16.459201 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:16.459168 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78467bdbc7-fw55x" Jul 21 00:29:16.459201 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:16.459168 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78467bdbc7-fw55x" event={"ID":"d9579e94-e396-4f13-a829-0829edc99f4d","Type":"ContainerDied","Data":"e8d52e83bed2f487b3e01d2335ec8e2671179ab5a2e0386f3376814537059904"} Jul 21 00:29:16.459418 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:16.459245 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78467bdbc7-fw55x" event={"ID":"d9579e94-e396-4f13-a829-0829edc99f4d","Type":"ContainerDied","Data":"666715bac384b9c48ff7fe7ddc0aeb3139398cd9298323288ddfab09f872e00e"} Jul 21 00:29:16.459418 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:16.459276 2580 scope.go:117] "RemoveContainer" containerID="e8d52e83bed2f487b3e01d2335ec8e2671179ab5a2e0386f3376814537059904" Jul 21 00:29:16.468444 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:16.468425 2580 scope.go:117] "RemoveContainer" containerID="e8d52e83bed2f487b3e01d2335ec8e2671179ab5a2e0386f3376814537059904" Jul 21 00:29:16.468845 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:29:16.468816 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d52e83bed2f487b3e01d2335ec8e2671179ab5a2e0386f3376814537059904\": container with ID starting with e8d52e83bed2f487b3e01d2335ec8e2671179ab5a2e0386f3376814537059904 not found: ID does not exist" containerID="e8d52e83bed2f487b3e01d2335ec8e2671179ab5a2e0386f3376814537059904" Jul 21 00:29:16.468944 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:16.468853 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d52e83bed2f487b3e01d2335ec8e2671179ab5a2e0386f3376814537059904"} err="failed to get container status \"e8d52e83bed2f487b3e01d2335ec8e2671179ab5a2e0386f3376814537059904\": rpc error: code = NotFound desc = could not find container \"e8d52e83bed2f487b3e01d2335ec8e2671179ab5a2e0386f3376814537059904\": container with ID starting with e8d52e83bed2f487b3e01d2335ec8e2671179ab5a2e0386f3376814537059904 not found: ID does not exist" Jul 21 00:29:16.476663 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:16.476637 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78467bdbc7-fw55x"] Jul 21 00:29:16.480577 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:16.480541 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-78467bdbc7-fw55x"] Jul 21 00:29:18.297398 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:18.297361 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9579e94-e396-4f13-a829-0829edc99f4d" path="/var/lib/kubelet/pods/d9579e94-e396-4f13-a829-0829edc99f4d/volumes" Jul 21 00:29:34.657755 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:34.657717 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:29:34.718027 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:34.717983 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:29:34.878326 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:34.878287 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:29:34.973545 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:34.973458 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:29:35.495383 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:35.495341 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:29:35.594182 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:35.594148 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-658c477b49-fw2r6"] Jul 21 00:29:35.594562 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:35.594549 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9579e94-e396-4f13-a829-0829edc99f4d" containerName="manager" Jul 21 00:29:35.594634 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:35.594564 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9579e94-e396-4f13-a829-0829edc99f4d" containerName="manager" Jul 21 00:29:35.594674 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:35.594644 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9579e94-e396-4f13-a829-0829edc99f4d" containerName="manager" Jul 21 00:29:35.597979 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:35.597959 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-658c477b49-fw2r6" Jul 21 00:29:35.600439 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:35.600414 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 21 00:29:35.601097 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:35.601080 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-572xg\"" Jul 21 00:29:35.613167 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:35.613141 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-658c477b49-fw2r6"] Jul 21 00:29:35.656492 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:35.656449 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa980c65-1c48-4872-bae1-082a7aeda718-openshift-service-ca\") pod \"maas-api-658c477b49-fw2r6\" (UID: \"fa980c65-1c48-4872-bae1-082a7aeda718\") " pod="odh-ai-gateway-infra/maas-api-658c477b49-fw2r6" Jul 21 00:29:35.656492 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:35.656493 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mscsq\" (UniqueName: \"kubernetes.io/projected/fa980c65-1c48-4872-bae1-082a7aeda718-kube-api-access-mscsq\") pod \"maas-api-658c477b49-fw2r6\" (UID: \"fa980c65-1c48-4872-bae1-082a7aeda718\") " pod="odh-ai-gateway-infra/maas-api-658c477b49-fw2r6" Jul 21 00:29:35.656789 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:35.656686 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa980c65-1c48-4872-bae1-082a7aeda718-maas-api-tls\") pod \"maas-api-658c477b49-fw2r6\" (UID: \"fa980c65-1c48-4872-bae1-082a7aeda718\") " pod="odh-ai-gateway-infra/maas-api-658c477b49-fw2r6" Jul 21 00:29:35.757248 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:35.757142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa980c65-1c48-4872-bae1-082a7aeda718-openshift-service-ca\") pod \"maas-api-658c477b49-fw2r6\" (UID: \"fa980c65-1c48-4872-bae1-082a7aeda718\") " pod="odh-ai-gateway-infra/maas-api-658c477b49-fw2r6" Jul 21 00:29:35.757248 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:35.757186 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mscsq\" (UniqueName: \"kubernetes.io/projected/fa980c65-1c48-4872-bae1-082a7aeda718-kube-api-access-mscsq\") pod \"maas-api-658c477b49-fw2r6\" (UID: \"fa980c65-1c48-4872-bae1-082a7aeda718\") " pod="odh-ai-gateway-infra/maas-api-658c477b49-fw2r6" Jul 21 00:29:35.757248 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:35.757247 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa980c65-1c48-4872-bae1-082a7aeda718-maas-api-tls\") pod \"maas-api-658c477b49-fw2r6\" (UID: \"fa980c65-1c48-4872-bae1-082a7aeda718\") " pod="odh-ai-gateway-infra/maas-api-658c477b49-fw2r6" Jul 21 00:29:35.757898 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:29:35.757347 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 21 00:29:35.757898 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:29:35.757419 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa980c65-1c48-4872-bae1-082a7aeda718-maas-api-tls podName:fa980c65-1c48-4872-bae1-082a7aeda718 nodeName:}" failed. No retries permitted until 2026-07-21 00:29:36.257401122 +0000 UTC m=+770.472781433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fa980c65-1c48-4872-bae1-082a7aeda718-maas-api-tls") pod "maas-api-658c477b49-fw2r6" (UID: "fa980c65-1c48-4872-bae1-082a7aeda718") : secret "maas-api-serving-cert" not found Jul 21 00:29:35.758016 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:35.757985 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa980c65-1c48-4872-bae1-082a7aeda718-openshift-service-ca\") pod \"maas-api-658c477b49-fw2r6\" (UID: \"fa980c65-1c48-4872-bae1-082a7aeda718\") " pod="odh-ai-gateway-infra/maas-api-658c477b49-fw2r6" Jul 21 00:29:35.767953 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:35.767926 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mscsq\" (UniqueName: \"kubernetes.io/projected/fa980c65-1c48-4872-bae1-082a7aeda718-kube-api-access-mscsq\") pod \"maas-api-658c477b49-fw2r6\" (UID: \"fa980c65-1c48-4872-bae1-082a7aeda718\") " pod="odh-ai-gateway-infra/maas-api-658c477b49-fw2r6" Jul 21 00:29:35.897698 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:35.897666 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:29:36.062199 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:36.062152 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:29:36.160921 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:36.160859 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:29:36.198260 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:36.198224 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-lxrfg"] Jul 21 00:29:36.204757 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:36.204732 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-lxrfg" Jul 21 00:29:36.207456 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:36.207430 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2lrhm\"" Jul 21 00:29:36.207632 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:36.207477 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 21 00:29:36.217372 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:36.217340 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-lxrfg"] Jul 21 00:29:36.262203 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:36.262168 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snknp\" (UniqueName: \"kubernetes.io/projected/9f3e5dde-da98-4b74-aed1-d9351b3e325c-kube-api-access-snknp\") pod \"payload-processing-5bf47cd988-lxrfg\" (UID: \"9f3e5dde-da98-4b74-aed1-d9351b3e325c\") " pod="openshift-ingress/payload-processing-5bf47cd988-lxrfg" Jul 21 00:29:36.262399 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:36.262235 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f3e5dde-da98-4b74-aed1-d9351b3e325c-plugins-config-volume\") pod \"payload-processing-5bf47cd988-lxrfg\" (UID: \"9f3e5dde-da98-4b74-aed1-d9351b3e325c\") " pod="openshift-ingress/payload-processing-5bf47cd988-lxrfg" Jul 21 00:29:36.262399 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:36.262271 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa980c65-1c48-4872-bae1-082a7aeda718-maas-api-tls\") pod \"maas-api-658c477b49-fw2r6\" (UID: \"fa980c65-1c48-4872-bae1-082a7aeda718\") " pod="odh-ai-gateway-infra/maas-api-658c477b49-fw2r6" Jul 21 00:29:36.264885 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:36.264861 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa980c65-1c48-4872-bae1-082a7aeda718-maas-api-tls\") pod \"maas-api-658c477b49-fw2r6\" (UID: \"fa980c65-1c48-4872-bae1-082a7aeda718\") " pod="odh-ai-gateway-infra/maas-api-658c477b49-fw2r6" Jul 21 00:29:36.362867 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:36.362786 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snknp\" (UniqueName: \"kubernetes.io/projected/9f3e5dde-da98-4b74-aed1-d9351b3e325c-kube-api-access-snknp\") pod \"payload-processing-5bf47cd988-lxrfg\" (UID: \"9f3e5dde-da98-4b74-aed1-d9351b3e325c\") " pod="openshift-ingress/payload-processing-5bf47cd988-lxrfg" Jul 21 00:29:36.362867 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:36.362840 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f3e5dde-da98-4b74-aed1-d9351b3e325c-plugins-config-volume\") pod \"payload-processing-5bf47cd988-lxrfg\" (UID: \"9f3e5dde-da98-4b74-aed1-d9351b3e325c\") " pod="openshift-ingress/payload-processing-5bf47cd988-lxrfg" Jul 21 00:29:36.363447 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:36.363428 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f3e5dde-da98-4b74-aed1-d9351b3e325c-plugins-config-volume\") pod \"payload-processing-5bf47cd988-lxrfg\" (UID: \"9f3e5dde-da98-4b74-aed1-d9351b3e325c\") " pod="openshift-ingress/payload-processing-5bf47cd988-lxrfg" Jul 21 00:29:36.380080 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:36.380048 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snknp\" (UniqueName: \"kubernetes.io/projected/9f3e5dde-da98-4b74-aed1-d9351b3e325c-kube-api-access-snknp\") pod \"payload-processing-5bf47cd988-lxrfg\" (UID: \"9f3e5dde-da98-4b74-aed1-d9351b3e325c\") " pod="openshift-ingress/payload-processing-5bf47cd988-lxrfg" Jul 21 00:29:36.508484 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:36.508442 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-658c477b49-fw2r6" Jul 21 00:29:36.517004 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:36.516974 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-lxrfg" Jul 21 00:29:36.691716 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:36.691155 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-lxrfg"] Jul 21 00:29:36.692796 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:29:36.692764 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f3e5dde_da98_4b74_aed1_d9351b3e325c.slice/crio-9ef3ab2a3f850b08c54c1ed61eb6ab0bce6520aebac32b8c20c06d0a0c399959 WatchSource:0}: Error finding container 9ef3ab2a3f850b08c54c1ed61eb6ab0bce6520aebac32b8c20c06d0a0c399959: Status 404 returned error can't find the container with id 9ef3ab2a3f850b08c54c1ed61eb6ab0bce6520aebac32b8c20c06d0a0c399959 Jul 21 00:29:36.711944 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:36.711920 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-658c477b49-fw2r6"] Jul 21 00:29:36.714118 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:29:36.714090 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa980c65_1c48_4872_bae1_082a7aeda718.slice/crio-bb3e3f9ce96649d9c0d6d311a47c3f8dec05060f93892bdc2f8e71865c355fe5 WatchSource:0}: Error finding container bb3e3f9ce96649d9c0d6d311a47c3f8dec05060f93892bdc2f8e71865c355fe5: Status 404 returned error can't find the container with id bb3e3f9ce96649d9c0d6d311a47c3f8dec05060f93892bdc2f8e71865c355fe5 Jul 21 00:29:37.578727 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:37.578628 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-658c477b49-fw2r6" event={"ID":"fa980c65-1c48-4872-bae1-082a7aeda718","Type":"ContainerStarted","Data":"bb3e3f9ce96649d9c0d6d311a47c3f8dec05060f93892bdc2f8e71865c355fe5"} Jul 21 00:29:37.580863 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:37.580780 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-lxrfg" event={"ID":"9f3e5dde-da98-4b74-aed1-d9351b3e325c","Type":"ContainerStarted","Data":"9ef3ab2a3f850b08c54c1ed61eb6ab0bce6520aebac32b8c20c06d0a0c399959"} Jul 21 00:29:39.600804 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:39.600759 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-658c477b49-fw2r6" event={"ID":"fa980c65-1c48-4872-bae1-082a7aeda718","Type":"ContainerStarted","Data":"cccb288de795ee35ead5160121f6e6b96f6d2f08142c6b3634e0e99f49cb7c09"} Jul 21 00:29:39.601303 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:39.600949 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-658c477b49-fw2r6" Jul 21 00:29:39.604041 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:39.604004 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-lxrfg" event={"ID":"9f3e5dde-da98-4b74-aed1-d9351b3e325c","Type":"ContainerStarted","Data":"88a3aab37215bd3bbb7f294a24309bd0163d115d849889720891eed24c414f02"} Jul 21 00:29:39.604806 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:39.604780 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-lxrfg" Jul 21 00:29:39.622425 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:39.622366 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-658c477b49-fw2r6" podStartSLOduration=2.392004583 podStartE2EDuration="4.622343567s" totalImagesPullingTime="2.230338984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:29:39.620489106 +0000 UTC m=+773.835869434" watchObservedRunningTime="2026-07-21 00:29:39.622343567 +0000 UTC m=+773.837723894" Jul 21 00:29:39.641194 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:39.641134 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-lxrfg" podStartSLOduration=1.384836311 podStartE2EDuration="3.641117915s" totalImagesPullingTime="2.256281604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:29:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:29:39.640441863 +0000 UTC m=+773.855822187" watchObservedRunningTime="2026-07-21 00:29:39.641117915 +0000 UTC m=+773.856498239" Jul 21 00:29:42.189853 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:42.189801 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-6vs2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 21 00:29:42.190253 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:42.189891 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" podUID="944990f8-eece-4bbe-a092-1db9b9547c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 21 00:29:44.636455 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:44.636423 2580 generic.go:356] "Generic (PLEG): container finished" podID="944990f8-eece-4bbe-a092-1db9b9547c2f" containerID="502f7e92bedd59183f7ca08000fe4a46b21a8c13e6b5b342bba2265a198c9f72" exitCode=0 Jul 21 00:29:44.636956 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:44.636492 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" event={"ID":"944990f8-eece-4bbe-a092-1db9b9547c2f","Type":"ContainerDied","Data":"502f7e92bedd59183f7ca08000fe4a46b21a8c13e6b5b342bba2265a198c9f72"} Jul 21 00:29:44.636956 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:44.636916 2580 scope.go:117] "RemoveContainer" containerID="502f7e92bedd59183f7ca08000fe4a46b21a8c13e6b5b342bba2265a198c9f72" Jul 21 00:29:44.637897 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:44.637866 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:29:44.638013 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:44.637953 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:29:44.638013 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:44.638003 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:29:45.619353 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:45.619324 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-658c477b49-fw2r6" Jul 21 00:29:45.644800 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:45.644762 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" event={"ID":"944990f8-eece-4bbe-a092-1db9b9547c2f","Type":"ContainerStarted","Data":"58bf22cc7a760c326c74385d96abb5efbe7adc6519b8c1c2b9a409eff1fca13d"} Jul 21 00:29:46.401446 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:46.400965 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:29:46.401446 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:46.401097 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-6vs2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 21 00:29:46.401446 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:46.401139 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" podUID="944990f8-eece-4bbe-a092-1db9b9547c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 21 00:29:46.683977 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:46.683865 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:29:46.752950 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:46.752916 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:29:47.400712 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:47.400656 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-6vs2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 21 00:29:47.400938 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:47.400726 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" podUID="944990f8-eece-4bbe-a092-1db9b9547c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 21 00:29:48.401006 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:48.400957 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-6vs2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 21 00:29:48.401519 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:48.401030 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" podUID="944990f8-eece-4bbe-a092-1db9b9547c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 21 00:29:49.400360 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:49.400306 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-6vs2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 21 00:29:49.400628 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:49.400414 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" podUID="944990f8-eece-4bbe-a092-1db9b9547c2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 21 00:29:50.404458 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:50.404429 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:29:50.404889 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:50.404810 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:29:50.405550 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:50.405535 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-6vs2d" Jul 21 00:29:51.622090 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:51.622059 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-lxrfg" Jul 21 00:29:57.089674 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:29:57.089632 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:30:00.137315 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:00.137274 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29743230-wddzw"] Jul 21 00:30:00.142229 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:00.142208 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743230-wddzw" Jul 21 00:30:00.144992 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:00.144967 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxv7p\"" Jul 21 00:30:00.148542 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:00.148507 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29743230-wddzw"] Jul 21 00:30:00.154276 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:00.154255 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743230-wddzw" Jul 21 00:30:00.295590 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:30:00.295555 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1bab934_69c0_4bfd_abe3_4961ff78b70a.slice/crio-210cf1e339d35ab8db06ba779432b3304a6d33ee29129d81b75f012507f78bcc WatchSource:0}: Error finding container 210cf1e339d35ab8db06ba779432b3304a6d33ee29129d81b75f012507f78bcc: Status 404 returned error can't find the container with id 210cf1e339d35ab8db06ba779432b3304a6d33ee29129d81b75f012507f78bcc Jul 21 00:30:00.299370 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:00.299344 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29743230-wddzw"] Jul 21 00:30:00.726748 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:00.726709 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743230-wddzw" event={"ID":"f1bab934-69c0-4bfd-abe3-4961ff78b70a","Type":"ContainerStarted","Data":"210cf1e339d35ab8db06ba779432b3304a6d33ee29129d81b75f012507f78bcc"} Jul 21 00:30:04.943241 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:04.943152 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl"] Jul 21 00:30:04.953292 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:04.953264 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" Jul 21 00:30:04.957849 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:04.957643 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 21 00:30:04.963023 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:04.959030 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-m5dwj\"" Jul 21 00:30:04.963023 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:04.959254 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 21 00:30:04.963023 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:04.959349 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 21 00:30:04.964273 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:04.964247 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl"] Jul 21 00:30:04.985453 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:04.985398 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:30:05.053911 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:05.053877 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51fa7b53-6a12-405e-aa14-30e3c86d7869-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mzvhl\" (UID: \"51fa7b53-6a12-405e-aa14-30e3c86d7869\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" Jul 21 00:30:05.054082 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:05.053971 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51fa7b53-6a12-405e-aa14-30e3c86d7869-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mzvhl\" (UID: \"51fa7b53-6a12-405e-aa14-30e3c86d7869\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" Jul 21 00:30:05.054082 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:05.054007 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51fa7b53-6a12-405e-aa14-30e3c86d7869-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mzvhl\" (UID: \"51fa7b53-6a12-405e-aa14-30e3c86d7869\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" Jul 21 00:30:05.054197 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:05.054102 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzbtl\" (UniqueName: \"kubernetes.io/projected/51fa7b53-6a12-405e-aa14-30e3c86d7869-kube-api-access-kzbtl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mzvhl\" (UID: \"51fa7b53-6a12-405e-aa14-30e3c86d7869\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" Jul 21 00:30:05.054197 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:05.054154 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51fa7b53-6a12-405e-aa14-30e3c86d7869-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mzvhl\" (UID: \"51fa7b53-6a12-405e-aa14-30e3c86d7869\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" Jul 21 00:30:05.054313 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:05.054197 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51fa7b53-6a12-405e-aa14-30e3c86d7869-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mzvhl\" (UID: \"51fa7b53-6a12-405e-aa14-30e3c86d7869\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" Jul 21 00:30:05.155434 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:05.155396 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51fa7b53-6a12-405e-aa14-30e3c86d7869-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mzvhl\" (UID: \"51fa7b53-6a12-405e-aa14-30e3c86d7869\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" Jul 21 00:30:05.155652 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:05.155452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51fa7b53-6a12-405e-aa14-30e3c86d7869-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mzvhl\" (UID: \"51fa7b53-6a12-405e-aa14-30e3c86d7869\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" Jul 21 00:30:05.155652 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:05.155521 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51fa7b53-6a12-405e-aa14-30e3c86d7869-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mzvhl\" (UID: \"51fa7b53-6a12-405e-aa14-30e3c86d7869\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" Jul 21 00:30:05.155652 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:05.155575 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51fa7b53-6a12-405e-aa14-30e3c86d7869-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mzvhl\" (UID: \"51fa7b53-6a12-405e-aa14-30e3c86d7869\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" Jul 21 00:30:05.155652 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:05.155641 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51fa7b53-6a12-405e-aa14-30e3c86d7869-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mzvhl\" (UID: \"51fa7b53-6a12-405e-aa14-30e3c86d7869\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" Jul 21 00:30:05.155879 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:05.155686 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzbtl\" (UniqueName: \"kubernetes.io/projected/51fa7b53-6a12-405e-aa14-30e3c86d7869-kube-api-access-kzbtl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mzvhl\" (UID: \"51fa7b53-6a12-405e-aa14-30e3c86d7869\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" Jul 21 00:30:05.156023 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:05.155999 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51fa7b53-6a12-405e-aa14-30e3c86d7869-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mzvhl\" (UID: \"51fa7b53-6a12-405e-aa14-30e3c86d7869\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" Jul 21 00:30:05.156312 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:05.156290 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51fa7b53-6a12-405e-aa14-30e3c86d7869-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mzvhl\" (UID: \"51fa7b53-6a12-405e-aa14-30e3c86d7869\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" Jul 21 00:30:05.156491 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:05.156473 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51fa7b53-6a12-405e-aa14-30e3c86d7869-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mzvhl\" (UID: \"51fa7b53-6a12-405e-aa14-30e3c86d7869\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" Jul 21 00:30:05.158208 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:05.158173 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51fa7b53-6a12-405e-aa14-30e3c86d7869-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mzvhl\" (UID: \"51fa7b53-6a12-405e-aa14-30e3c86d7869\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" Jul 21 00:30:05.158823 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:05.158798 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51fa7b53-6a12-405e-aa14-30e3c86d7869-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mzvhl\" (UID: \"51fa7b53-6a12-405e-aa14-30e3c86d7869\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" Jul 21 00:30:05.165644 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:05.165579 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzbtl\" (UniqueName: \"kubernetes.io/projected/51fa7b53-6a12-405e-aa14-30e3c86d7869-kube-api-access-kzbtl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mzvhl\" (UID: \"51fa7b53-6a12-405e-aa14-30e3c86d7869\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" Jul 21 00:30:05.269201 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:05.269153 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" Jul 21 00:30:05.541640 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:05.541612 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl"] Jul 21 00:30:05.544154 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:30:05.544123 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fa7b53_6a12_405e_aa14_30e3c86d7869.slice/crio-f3b82e7a2e275c8c185443bfbc5aee00ed119f143fd8bee4e9e4e670c29d2316 WatchSource:0}: Error finding container f3b82e7a2e275c8c185443bfbc5aee00ed119f143fd8bee4e9e4e670c29d2316: Status 404 returned error can't find the container with id f3b82e7a2e275c8c185443bfbc5aee00ed119f143fd8bee4e9e4e670c29d2316 Jul 21 00:30:05.755411 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:05.755370 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743230-wddzw" event={"ID":"f1bab934-69c0-4bfd-abe3-4961ff78b70a","Type":"ContainerStarted","Data":"77e02ce9efb4313efc55427186104e40632bb1f98ed34704a216ac630042938d"} Jul 21 00:30:05.756567 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:05.756543 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" event={"ID":"51fa7b53-6a12-405e-aa14-30e3c86d7869","Type":"ContainerStarted","Data":"f3b82e7a2e275c8c185443bfbc5aee00ed119f143fd8bee4e9e4e670c29d2316"} Jul 21 00:30:05.805966 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:05.805895 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743230-wddzw" podStartSLOduration=0.636643398 podStartE2EDuration="5.805871887s" totalImagesPullingTime="5.169228489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:30:05.803288864 +0000 UTC m=+800.018669188" watchObservedRunningTime="2026-07-21 00:30:05.805871887 +0000 UTC m=+800.021252212" Jul 21 00:30:06.763679 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:06.763645 2580 generic.go:356] "Generic (PLEG): container finished" podID="f1bab934-69c0-4bfd-abe3-4961ff78b70a" containerID="77e02ce9efb4313efc55427186104e40632bb1f98ed34704a216ac630042938d" exitCode=0 Jul 21 00:30:06.764071 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:06.763728 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743230-wddzw" event={"ID":"f1bab934-69c0-4bfd-abe3-4961ff78b70a","Type":"ContainerDied","Data":"77e02ce9efb4313efc55427186104e40632bb1f98ed34704a216ac630042938d"} Jul 21 00:30:07.772393 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:07.772354 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" event={"ID":"51fa7b53-6a12-405e-aa14-30e3c86d7869","Type":"ContainerStarted","Data":"97c07da7bc07ded949ab33dbcee9c919bce6e685c3e1cc61d6cd622f9e6174cf"} Jul 21 00:30:07.792905 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:07.792841 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" podStartSLOduration=2.541767096 podStartE2EDuration="3.792823776s" totalImagesPullingTime="1.25105668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:30:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:30:07.790378958 +0000 UTC m=+802.005759286" watchObservedRunningTime="2026-07-21 00:30:07.792823776 +0000 UTC m=+802.008204100" Jul 21 00:30:07.911807 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:07.911781 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743230-wddzw" Jul 21 00:30:08.781647 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:08.781536 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743230-wddzw" Jul 21 00:30:08.781647 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:08.781573 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743230-wddzw" event={"ID":"f1bab934-69c0-4bfd-abe3-4961ff78b70a","Type":"ContainerDied","Data":"210cf1e339d35ab8db06ba779432b3304a6d33ee29129d81b75f012507f78bcc"} Jul 21 00:30:08.781647 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:08.781640 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="210cf1e339d35ab8db06ba779432b3304a6d33ee29129d81b75f012507f78bcc" Jul 21 00:30:15.269532 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:15.269487 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" Jul 21 00:30:15.269980 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:15.269558 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" Jul 21 00:30:15.282245 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:15.282217 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" Jul 21 00:30:15.830168 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:15.830135 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mzvhl" Jul 21 00:30:19.350307 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.350271 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn"] Jul 21 00:30:19.350840 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.350703 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1bab934-69c0-4bfd-abe3-4961ff78b70a" containerName="cleanup" Jul 21 00:30:19.350840 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.350717 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1bab934-69c0-4bfd-abe3-4961ff78b70a" containerName="cleanup" Jul 21 00:30:19.350840 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.350801 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1bab934-69c0-4bfd-abe3-4961ff78b70a" containerName="cleanup" Jul 21 00:30:19.354990 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.354966 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" Jul 21 00:30:19.359520 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.359498 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 21 00:30:19.367699 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.367668 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn"] Jul 21 00:30:19.397540 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.397509 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:30:19.490111 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.490061 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4290eeee-cec8-49d5-b4cc-a02807ed098f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lgvkn\" (UID: \"4290eeee-cec8-49d5-b4cc-a02807ed098f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" Jul 21 00:30:19.490288 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.490192 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4290eeee-cec8-49d5-b4cc-a02807ed098f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lgvkn\" (UID: \"4290eeee-cec8-49d5-b4cc-a02807ed098f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" Jul 21 00:30:19.490288 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.490248 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4290eeee-cec8-49d5-b4cc-a02807ed098f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lgvkn\" (UID: \"4290eeee-cec8-49d5-b4cc-a02807ed098f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" Jul 21 00:30:19.490377 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.490330 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4290eeee-cec8-49d5-b4cc-a02807ed098f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lgvkn\" (UID: \"4290eeee-cec8-49d5-b4cc-a02807ed098f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" Jul 21 00:30:19.490377 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.490364 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4290eeee-cec8-49d5-b4cc-a02807ed098f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lgvkn\" (UID: \"4290eeee-cec8-49d5-b4cc-a02807ed098f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" Jul 21 00:30:19.490472 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.490390 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxh27\" (UniqueName: \"kubernetes.io/projected/4290eeee-cec8-49d5-b4cc-a02807ed098f-kube-api-access-kxh27\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lgvkn\" (UID: \"4290eeee-cec8-49d5-b4cc-a02807ed098f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" Jul 21 00:30:19.591349 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.591302 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4290eeee-cec8-49d5-b4cc-a02807ed098f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lgvkn\" (UID: \"4290eeee-cec8-49d5-b4cc-a02807ed098f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" Jul 21 00:30:19.591349 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.591353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4290eeee-cec8-49d5-b4cc-a02807ed098f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lgvkn\" (UID: \"4290eeee-cec8-49d5-b4cc-a02807ed098f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" Jul 21 00:30:19.591657 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.591379 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxh27\" (UniqueName: \"kubernetes.io/projected/4290eeee-cec8-49d5-b4cc-a02807ed098f-kube-api-access-kxh27\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lgvkn\" (UID: \"4290eeee-cec8-49d5-b4cc-a02807ed098f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" Jul 21 00:30:19.591657 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.591513 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4290eeee-cec8-49d5-b4cc-a02807ed098f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lgvkn\" (UID: \"4290eeee-cec8-49d5-b4cc-a02807ed098f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" Jul 21 00:30:19.591657 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.591648 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4290eeee-cec8-49d5-b4cc-a02807ed098f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lgvkn\" (UID: \"4290eeee-cec8-49d5-b4cc-a02807ed098f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" Jul 21 00:30:19.591819 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.591728 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4290eeee-cec8-49d5-b4cc-a02807ed098f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lgvkn\" (UID: \"4290eeee-cec8-49d5-b4cc-a02807ed098f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" Jul 21 00:30:19.591819 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.591770 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4290eeee-cec8-49d5-b4cc-a02807ed098f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lgvkn\" (UID: \"4290eeee-cec8-49d5-b4cc-a02807ed098f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" Jul 21 00:30:19.591924 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.591830 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4290eeee-cec8-49d5-b4cc-a02807ed098f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lgvkn\" (UID: \"4290eeee-cec8-49d5-b4cc-a02807ed098f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" Jul 21 00:30:19.592042 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.592023 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4290eeee-cec8-49d5-b4cc-a02807ed098f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lgvkn\" (UID: \"4290eeee-cec8-49d5-b4cc-a02807ed098f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" Jul 21 00:30:19.593838 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.593815 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4290eeee-cec8-49d5-b4cc-a02807ed098f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lgvkn\" (UID: \"4290eeee-cec8-49d5-b4cc-a02807ed098f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" Jul 21 00:30:19.594229 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.594212 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4290eeee-cec8-49d5-b4cc-a02807ed098f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lgvkn\" (UID: \"4290eeee-cec8-49d5-b4cc-a02807ed098f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" Jul 21 00:30:19.599612 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.599570 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxh27\" (UniqueName: \"kubernetes.io/projected/4290eeee-cec8-49d5-b4cc-a02807ed098f-kube-api-access-kxh27\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lgvkn\" (UID: \"4290eeee-cec8-49d5-b4cc-a02807ed098f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" Jul 21 00:30:19.665784 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.665681 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" Jul 21 00:30:19.810895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.810872 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn"] Jul 21 00:30:19.813311 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:30:19.813271 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4290eeee_cec8_49d5_b4cc_a02807ed098f.slice/crio-2fe7bcd5fc4f9723c4f86876171f099297e1a9c94bb9096f687c4c78ef7d2cfc WatchSource:0}: Error finding container 2fe7bcd5fc4f9723c4f86876171f099297e1a9c94bb9096f687c4c78ef7d2cfc: Status 404 returned error can't find the container with id 2fe7bcd5fc4f9723c4f86876171f099297e1a9c94bb9096f687c4c78ef7d2cfc Jul 21 00:30:19.842061 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:19.842030 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" event={"ID":"4290eeee-cec8-49d5-b4cc-a02807ed098f","Type":"ContainerStarted","Data":"2fe7bcd5fc4f9723c4f86876171f099297e1a9c94bb9096f687c4c78ef7d2cfc"} Jul 21 00:30:20.851196 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:20.851156 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" event={"ID":"4290eeee-cec8-49d5-b4cc-a02807ed098f","Type":"ContainerStarted","Data":"67ef0924e9d4d73a18213b03bd96a308817e7afe99782f25578498cb58cc0574"} Jul 21 00:30:20.873027 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:20.872962 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" podStartSLOduration=1.671959481 podStartE2EDuration="1.872939568s" totalImagesPullingTime="200.980087ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:30:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:30:20.871110053 +0000 UTC m=+815.086490394" watchObservedRunningTime="2026-07-21 00:30:20.872939568 +0000 UTC m=+815.088319894" Jul 21 00:30:29.665931 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:29.665874 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" Jul 21 00:30:29.666355 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:29.665981 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" Jul 21 00:30:29.678664 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:29.678634 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" Jul 21 00:30:29.913107 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:29.913068 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lgvkn" Jul 21 00:30:31.491426 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:31.491392 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:30:46.884561 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:30:46.884501 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:31:01.456445 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.456412 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7"] Jul 21 00:31:01.461507 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.461478 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" Jul 21 00:31:01.464255 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.464232 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 21 00:31:01.470151 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.470125 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7"] Jul 21 00:31:01.491133 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.491089 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:31:01.591471 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.591436 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc686d9a-c5b8-4792-ac40-99ac7151ad49-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7\" (UID: \"bc686d9a-c5b8-4792-ac40-99ac7151ad49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" Jul 21 00:31:01.591681 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.591485 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l69tq\" (UniqueName: \"kubernetes.io/projected/bc686d9a-c5b8-4792-ac40-99ac7151ad49-kube-api-access-l69tq\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7\" (UID: \"bc686d9a-c5b8-4792-ac40-99ac7151ad49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" Jul 21 00:31:01.591681 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.591588 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc686d9a-c5b8-4792-ac40-99ac7151ad49-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7\" (UID: \"bc686d9a-c5b8-4792-ac40-99ac7151ad49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" Jul 21 00:31:01.591681 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.591659 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc686d9a-c5b8-4792-ac40-99ac7151ad49-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7\" (UID: \"bc686d9a-c5b8-4792-ac40-99ac7151ad49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" Jul 21 00:31:01.591794 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.591693 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc686d9a-c5b8-4792-ac40-99ac7151ad49-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7\" (UID: \"bc686d9a-c5b8-4792-ac40-99ac7151ad49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" Jul 21 00:31:01.591794 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.591755 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc686d9a-c5b8-4792-ac40-99ac7151ad49-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7\" (UID: \"bc686d9a-c5b8-4792-ac40-99ac7151ad49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" Jul 21 00:31:01.692476 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.692443 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc686d9a-c5b8-4792-ac40-99ac7151ad49-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7\" (UID: \"bc686d9a-c5b8-4792-ac40-99ac7151ad49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" Jul 21 00:31:01.692723 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.692489 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l69tq\" (UniqueName: \"kubernetes.io/projected/bc686d9a-c5b8-4792-ac40-99ac7151ad49-kube-api-access-l69tq\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7\" (UID: \"bc686d9a-c5b8-4792-ac40-99ac7151ad49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" Jul 21 00:31:01.692723 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.692555 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc686d9a-c5b8-4792-ac40-99ac7151ad49-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7\" (UID: \"bc686d9a-c5b8-4792-ac40-99ac7151ad49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" Jul 21 00:31:01.692723 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.692577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc686d9a-c5b8-4792-ac40-99ac7151ad49-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7\" (UID: \"bc686d9a-c5b8-4792-ac40-99ac7151ad49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" Jul 21 00:31:01.692723 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.692622 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc686d9a-c5b8-4792-ac40-99ac7151ad49-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7\" (UID: \"bc686d9a-c5b8-4792-ac40-99ac7151ad49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" Jul 21 00:31:01.692723 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.692662 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc686d9a-c5b8-4792-ac40-99ac7151ad49-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7\" (UID: \"bc686d9a-c5b8-4792-ac40-99ac7151ad49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" Jul 21 00:31:01.693010 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.692976 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc686d9a-c5b8-4792-ac40-99ac7151ad49-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7\" (UID: \"bc686d9a-c5b8-4792-ac40-99ac7151ad49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" Jul 21 00:31:01.693060 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.693012 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc686d9a-c5b8-4792-ac40-99ac7151ad49-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7\" (UID: \"bc686d9a-c5b8-4792-ac40-99ac7151ad49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" Jul 21 00:31:01.693060 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.693049 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc686d9a-c5b8-4792-ac40-99ac7151ad49-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7\" (UID: \"bc686d9a-c5b8-4792-ac40-99ac7151ad49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" Jul 21 00:31:01.694968 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.694950 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc686d9a-c5b8-4792-ac40-99ac7151ad49-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7\" (UID: \"bc686d9a-c5b8-4792-ac40-99ac7151ad49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" Jul 21 00:31:01.695243 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.695226 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc686d9a-c5b8-4792-ac40-99ac7151ad49-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7\" (UID: \"bc686d9a-c5b8-4792-ac40-99ac7151ad49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" Jul 21 00:31:01.701868 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.701843 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l69tq\" (UniqueName: \"kubernetes.io/projected/bc686d9a-c5b8-4792-ac40-99ac7151ad49-kube-api-access-l69tq\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7\" (UID: \"bc686d9a-c5b8-4792-ac40-99ac7151ad49\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" Jul 21 00:31:01.772714 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.772593 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" Jul 21 00:31:01.913344 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:01.913316 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7"] Jul 21 00:31:01.915065 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:31:01.915034 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc686d9a_c5b8_4792_ac40_99ac7151ad49.slice/crio-b3ece023c74d6bceee0232bb534a47df3c28ec117c470171b3d8d6538f03194a WatchSource:0}: Error finding container b3ece023c74d6bceee0232bb534a47df3c28ec117c470171b3d8d6538f03194a: Status 404 returned error can't find the container with id b3ece023c74d6bceee0232bb534a47df3c28ec117c470171b3d8d6538f03194a Jul 21 00:31:02.081393 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:02.081357 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" event={"ID":"bc686d9a-c5b8-4792-ac40-99ac7151ad49","Type":"ContainerStarted","Data":"b3ece023c74d6bceee0232bb534a47df3c28ec117c470171b3d8d6538f03194a"} Jul 21 00:31:03.088351 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:03.088312 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" event={"ID":"bc686d9a-c5b8-4792-ac40-99ac7151ad49","Type":"ContainerStarted","Data":"9fd2d31a5dfd4df468bfb3c1cba83f7200fdbfebbb825ac284757308e7c3391f"} Jul 21 00:31:03.109195 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:03.109141 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" podStartSLOduration=1.916511013 podStartE2EDuration="2.109123755s" totalImagesPullingTime="192.612742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:31:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:31:03.105827187 +0000 UTC m=+857.321207512" watchObservedRunningTime="2026-07-21 00:31:03.109123755 +0000 UTC m=+857.324504079" Jul 21 00:31:11.773676 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:11.773631 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" Jul 21 00:31:11.774162 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:11.773694 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" Jul 21 00:31:11.786018 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:11.785980 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" Jul 21 00:31:12.149726 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:12.149690 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7" Jul 21 00:31:27.870832 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:27.870788 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:31:27.971615 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:31:27.971565 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:32:05.650667 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:05.650625 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:32:05.755003 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:05.754968 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:32:06.817689 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:06.817646 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:32:06.825670 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:06.825635 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f755bfcdc-qqx9g"] Jul 21 00:32:06.825932 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:06.825907 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7f755bfcdc-qqx9g" podUID="20161088-5d29-4f83-b8d8-0a4292c9d959" containerName="authorino" containerID="cri-o://e6e950adf727ed9e706057dd736889d4da3129f4fca7050ecb78c6803f0329ad" gracePeriod=30 Jul 21 00:32:06.932389 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:06.932348 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:32:07.091576 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:07.091546 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f755bfcdc-qqx9g" Jul 21 00:32:07.113943 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:07.113903 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brqbz\" (UniqueName: \"kubernetes.io/projected/20161088-5d29-4f83-b8d8-0a4292c9d959-kube-api-access-brqbz\") pod \"20161088-5d29-4f83-b8d8-0a4292c9d959\" (UID: \"20161088-5d29-4f83-b8d8-0a4292c9d959\") " Jul 21 00:32:07.114126 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:07.114005 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/20161088-5d29-4f83-b8d8-0a4292c9d959-tls-cert\") pod \"20161088-5d29-4f83-b8d8-0a4292c9d959\" (UID: \"20161088-5d29-4f83-b8d8-0a4292c9d959\") " Jul 21 00:32:07.116227 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:07.116193 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20161088-5d29-4f83-b8d8-0a4292c9d959-kube-api-access-brqbz" (OuterVolumeSpecName: "kube-api-access-brqbz") pod "20161088-5d29-4f83-b8d8-0a4292c9d959" (UID: "20161088-5d29-4f83-b8d8-0a4292c9d959"). InnerVolumeSpecName "kube-api-access-brqbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:32:07.127225 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:07.127180 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20161088-5d29-4f83-b8d8-0a4292c9d959-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "20161088-5d29-4f83-b8d8-0a4292c9d959" (UID: "20161088-5d29-4f83-b8d8-0a4292c9d959"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:32:07.214975 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:07.214931 2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/20161088-5d29-4f83-b8d8-0a4292c9d959-tls-cert\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:32:07.214975 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:07.214966 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brqbz\" (UniqueName: \"kubernetes.io/projected/20161088-5d29-4f83-b8d8-0a4292c9d959-kube-api-access-brqbz\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:32:07.459731 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:07.459643 2580 generic.go:356] "Generic (PLEG): container finished" podID="20161088-5d29-4f83-b8d8-0a4292c9d959" containerID="e6e950adf727ed9e706057dd736889d4da3129f4fca7050ecb78c6803f0329ad" exitCode=0 Jul 21 00:32:07.459731 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:07.459712 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f755bfcdc-qqx9g" event={"ID":"20161088-5d29-4f83-b8d8-0a4292c9d959","Type":"ContainerDied","Data":"e6e950adf727ed9e706057dd736889d4da3129f4fca7050ecb78c6803f0329ad"} Jul 21 00:32:07.459919 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:07.459754 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f755bfcdc-qqx9g" Jul 21 00:32:07.459919 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:07.459766 2580 scope.go:117] "RemoveContainer" containerID="e6e950adf727ed9e706057dd736889d4da3129f4fca7050ecb78c6803f0329ad" Jul 21 00:32:07.459919 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:07.459754 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f755bfcdc-qqx9g" event={"ID":"20161088-5d29-4f83-b8d8-0a4292c9d959","Type":"ContainerDied","Data":"ed9d1e58721878bd19f7481f7eeea80798130efe4e34229df6569187c4dc7a3d"} Jul 21 00:32:07.469802 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:07.469783 2580 scope.go:117] "RemoveContainer" containerID="e6e950adf727ed9e706057dd736889d4da3129f4fca7050ecb78c6803f0329ad" Jul 21 00:32:07.470098 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:32:07.470075 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6e950adf727ed9e706057dd736889d4da3129f4fca7050ecb78c6803f0329ad\": container with ID starting with e6e950adf727ed9e706057dd736889d4da3129f4fca7050ecb78c6803f0329ad not found: ID does not exist" containerID="e6e950adf727ed9e706057dd736889d4da3129f4fca7050ecb78c6803f0329ad" Jul 21 00:32:07.470209 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:07.470107 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e950adf727ed9e706057dd736889d4da3129f4fca7050ecb78c6803f0329ad"} err="failed to get container status \"e6e950adf727ed9e706057dd736889d4da3129f4fca7050ecb78c6803f0329ad\": rpc error: code = NotFound desc = could not find container \"e6e950adf727ed9e706057dd736889d4da3129f4fca7050ecb78c6803f0329ad\": container with ID starting with e6e950adf727ed9e706057dd736889d4da3129f4fca7050ecb78c6803f0329ad not found: ID does not exist" Jul 21 00:32:07.483923 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:07.483889 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f755bfcdc-qqx9g"] Jul 21 00:32:07.486516 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:07.486483 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7f755bfcdc-qqx9g"] Jul 21 00:32:08.298289 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:08.298200 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20161088-5d29-4f83-b8d8-0a4292c9d959" path="/var/lib/kubelet/pods/20161088-5d29-4f83-b8d8-0a4292c9d959/volumes" Jul 21 00:32:10.088456 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:10.088416 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:32:10.153568 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:10.153532 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:32:41.695908 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:32:41.695857 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:33:54.814910 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:33:54.814867 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-79bfd5d9f6-vhmvr"] Jul 21 00:33:54.815423 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:33:54.815265 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20161088-5d29-4f83-b8d8-0a4292c9d959" containerName="authorino" Jul 21 00:33:54.815423 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:33:54.815276 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="20161088-5d29-4f83-b8d8-0a4292c9d959" containerName="authorino" Jul 21 00:33:54.815423 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:33:54.815343 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="20161088-5d29-4f83-b8d8-0a4292c9d959" containerName="authorino" Jul 21 00:33:54.818386 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:33:54.818366 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79bfd5d9f6-vhmvr" Jul 21 00:33:54.820912 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:33:54.820885 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nhqrc\"" Jul 21 00:33:54.821052 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:33:54.820885 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 00:33:54.821052 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:33:54.820886 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 00:33:54.828542 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:33:54.828509 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79bfd5d9f6-vhmvr"] Jul 21 00:33:54.915491 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:33:54.915453 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wbj2\" (UniqueName: \"kubernetes.io/projected/2e92109f-5254-425e-8287-86b87001f2c8-kube-api-access-2wbj2\") pod \"maas-controller-79bfd5d9f6-vhmvr\" (UID: \"2e92109f-5254-425e-8287-86b87001f2c8\") " pod="opendatahub/maas-controller-79bfd5d9f6-vhmvr" Jul 21 00:33:54.915699 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:33:54.915531 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e92109f-5254-425e-8287-86b87001f2c8-webhook-cert\") pod \"maas-controller-79bfd5d9f6-vhmvr\" (UID: \"2e92109f-5254-425e-8287-86b87001f2c8\") " pod="opendatahub/maas-controller-79bfd5d9f6-vhmvr" Jul 21 00:33:55.016628 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:33:55.016557 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wbj2\" (UniqueName: \"kubernetes.io/projected/2e92109f-5254-425e-8287-86b87001f2c8-kube-api-access-2wbj2\") pod \"maas-controller-79bfd5d9f6-vhmvr\" (UID: \"2e92109f-5254-425e-8287-86b87001f2c8\") " pod="opendatahub/maas-controller-79bfd5d9f6-vhmvr" Jul 21 00:33:55.016808 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:33:55.016652 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e92109f-5254-425e-8287-86b87001f2c8-webhook-cert\") pod \"maas-controller-79bfd5d9f6-vhmvr\" (UID: \"2e92109f-5254-425e-8287-86b87001f2c8\") " pod="opendatahub/maas-controller-79bfd5d9f6-vhmvr" Jul 21 00:33:55.019227 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:33:55.019206 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e92109f-5254-425e-8287-86b87001f2c8-webhook-cert\") pod \"maas-controller-79bfd5d9f6-vhmvr\" (UID: \"2e92109f-5254-425e-8287-86b87001f2c8\") " pod="opendatahub/maas-controller-79bfd5d9f6-vhmvr" Jul 21 00:33:55.025373 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:33:55.025344 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wbj2\" (UniqueName: \"kubernetes.io/projected/2e92109f-5254-425e-8287-86b87001f2c8-kube-api-access-2wbj2\") pod \"maas-controller-79bfd5d9f6-vhmvr\" (UID: \"2e92109f-5254-425e-8287-86b87001f2c8\") " pod="opendatahub/maas-controller-79bfd5d9f6-vhmvr" Jul 21 00:33:55.130516 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:33:55.130430 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79bfd5d9f6-vhmvr" Jul 21 00:33:55.282309 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:33:55.282286 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79bfd5d9f6-vhmvr"] Jul 21 00:33:55.285346 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:33:55.285317 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e92109f_5254_425e_8287_86b87001f2c8.slice/crio-26651d414b98e3fbe5f8c7fd524ea5e6609eb663a2405951a48cce7e358f6d8c WatchSource:0}: Error finding container 26651d414b98e3fbe5f8c7fd524ea5e6609eb663a2405951a48cce7e358f6d8c: Status 404 returned error can't find the container with id 26651d414b98e3fbe5f8c7fd524ea5e6609eb663a2405951a48cce7e358f6d8c Jul 21 00:33:55.286721 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:33:55.286694 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 00:33:56.087214 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:33:56.087174 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79bfd5d9f6-vhmvr" event={"ID":"2e92109f-5254-425e-8287-86b87001f2c8","Type":"ContainerStarted","Data":"34529037beb37e04703ad471d2ab4f98e7a87773d72e89df7e774869a646c186"} Jul 21 00:33:56.087713 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:33:56.087244 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79bfd5d9f6-vhmvr" event={"ID":"2e92109f-5254-425e-8287-86b87001f2c8","Type":"ContainerStarted","Data":"26651d414b98e3fbe5f8c7fd524ea5e6609eb663a2405951a48cce7e358f6d8c"} Jul 21 00:33:56.087713 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:33:56.087277 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-79bfd5d9f6-vhmvr" Jul 21 00:33:56.103583 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:33:56.103524 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-79bfd5d9f6-vhmvr" podStartSLOduration=1.708122431 podStartE2EDuration="2.103507898s" totalImagesPullingTime="395.385467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:33:56.101883331 +0000 UTC m=+1030.317263657" watchObservedRunningTime="2026-07-21 00:33:56.103507898 +0000 UTC m=+1030.318888222" Jul 21 00:34:07.095834 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:07.095802 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-79bfd5d9f6-vhmvr" Jul 21 00:34:11.646182 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:11.646140 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:34:11.753559 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:11.753516 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-79bfd5d9f6-vhmvr"] Jul 21 00:34:11.753852 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:11.753815 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-79bfd5d9f6-vhmvr" podUID="2e92109f-5254-425e-8287-86b87001f2c8" containerName="manager" containerID="cri-o://34529037beb37e04703ad471d2ab4f98e7a87773d72e89df7e774869a646c186" gracePeriod=30 Jul 21 00:34:12.003577 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.003545 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79bfd5d9f6-vhmvr" Jul 21 00:34:12.178499 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.178408 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e92109f-5254-425e-8287-86b87001f2c8-webhook-cert\") pod \"2e92109f-5254-425e-8287-86b87001f2c8\" (UID: \"2e92109f-5254-425e-8287-86b87001f2c8\") " Jul 21 00:34:12.178499 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.178457 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wbj2\" (UniqueName: \"kubernetes.io/projected/2e92109f-5254-425e-8287-86b87001f2c8-kube-api-access-2wbj2\") pod \"2e92109f-5254-425e-8287-86b87001f2c8\" (UID: \"2e92109f-5254-425e-8287-86b87001f2c8\") " Jul 21 00:34:12.180893 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.180849 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e92109f-5254-425e-8287-86b87001f2c8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2e92109f-5254-425e-8287-86b87001f2c8" (UID: "2e92109f-5254-425e-8287-86b87001f2c8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:34:12.180893 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.180880 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e92109f-5254-425e-8287-86b87001f2c8-kube-api-access-2wbj2" (OuterVolumeSpecName: "kube-api-access-2wbj2") pod "2e92109f-5254-425e-8287-86b87001f2c8" (UID: "2e92109f-5254-425e-8287-86b87001f2c8"). InnerVolumeSpecName "kube-api-access-2wbj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:34:12.188560 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.188528 2580 generic.go:356] "Generic (PLEG): container finished" podID="2e92109f-5254-425e-8287-86b87001f2c8" containerID="34529037beb37e04703ad471d2ab4f98e7a87773d72e89df7e774869a646c186" exitCode=0 Jul 21 00:34:12.188719 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.188573 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79bfd5d9f6-vhmvr" Jul 21 00:34:12.188719 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.188619 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79bfd5d9f6-vhmvr" event={"ID":"2e92109f-5254-425e-8287-86b87001f2c8","Type":"ContainerDied","Data":"34529037beb37e04703ad471d2ab4f98e7a87773d72e89df7e774869a646c186"} Jul 21 00:34:12.188719 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.188681 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79bfd5d9f6-vhmvr" event={"ID":"2e92109f-5254-425e-8287-86b87001f2c8","Type":"ContainerDied","Data":"26651d414b98e3fbe5f8c7fd524ea5e6609eb663a2405951a48cce7e358f6d8c"} Jul 21 00:34:12.188719 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.188706 2580 scope.go:117] "RemoveContainer" containerID="34529037beb37e04703ad471d2ab4f98e7a87773d72e89df7e774869a646c186" Jul 21 00:34:12.198513 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.198493 2580 scope.go:117] "RemoveContainer" containerID="34529037beb37e04703ad471d2ab4f98e7a87773d72e89df7e774869a646c186" Jul 21 00:34:12.198877 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:34:12.198847 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34529037beb37e04703ad471d2ab4f98e7a87773d72e89df7e774869a646c186\": container with ID starting with 34529037beb37e04703ad471d2ab4f98e7a87773d72e89df7e774869a646c186 not found: ID does not exist" containerID="34529037beb37e04703ad471d2ab4f98e7a87773d72e89df7e774869a646c186" Jul 21 00:34:12.198959 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.198892 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34529037beb37e04703ad471d2ab4f98e7a87773d72e89df7e774869a646c186"} err="failed to get container status \"34529037beb37e04703ad471d2ab4f98e7a87773d72e89df7e774869a646c186\": rpc error: code = NotFound desc = could not find container \"34529037beb37e04703ad471d2ab4f98e7a87773d72e89df7e774869a646c186\": container with ID starting with 34529037beb37e04703ad471d2ab4f98e7a87773d72e89df7e774869a646c186 not found: ID does not exist" Jul 21 00:34:12.210741 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.210707 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-79bfd5d9f6-vhmvr"] Jul 21 00:34:12.213056 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.213032 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-79bfd5d9f6-vhmvr"] Jul 21 00:34:12.279638 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.279574 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e92109f-5254-425e-8287-86b87001f2c8-webhook-cert\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:34:12.279638 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.279638 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wbj2\" (UniqueName: \"kubernetes.io/projected/2e92109f-5254-425e-8287-86b87001f2c8-kube-api-access-2wbj2\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:34:12.298031 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.297998 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e92109f-5254-425e-8287-86b87001f2c8" path="/var/lib/kubelet/pods/2e92109f-5254-425e-8287-86b87001f2c8/volumes" Jul 21 00:34:12.862426 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.862390 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:34:12.987625 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.987559 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-79bfd5d9f6-d9ft9"] Jul 21 00:34:12.988503 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.988466 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e92109f-5254-425e-8287-86b87001f2c8" containerName="manager" Jul 21 00:34:12.988503 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.988502 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e92109f-5254-425e-8287-86b87001f2c8" containerName="manager" Jul 21 00:34:12.988758 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.988744 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e92109f-5254-425e-8287-86b87001f2c8" containerName="manager" Jul 21 00:34:12.994398 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.994297 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79bfd5d9f6-d9ft9" Jul 21 00:34:12.995820 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.995795 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79bfd5d9f6-d9ft9"] Jul 21 00:34:12.998628 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.998315 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 21 00:34:12.998628 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.998350 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nhqrc\"" Jul 21 00:34:12.998628 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:12.998393 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 21 00:34:13.091594 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:13.091548 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pxzt\" (UniqueName: \"kubernetes.io/projected/d75f19e5-a207-4df5-a72e-9782ba3927a8-kube-api-access-6pxzt\") pod \"maas-controller-79bfd5d9f6-d9ft9\" (UID: \"d75f19e5-a207-4df5-a72e-9782ba3927a8\") " pod="opendatahub/maas-controller-79bfd5d9f6-d9ft9" Jul 21 00:34:13.091837 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:13.091717 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d75f19e5-a207-4df5-a72e-9782ba3927a8-webhook-cert\") pod \"maas-controller-79bfd5d9f6-d9ft9\" (UID: \"d75f19e5-a207-4df5-a72e-9782ba3927a8\") " pod="opendatahub/maas-controller-79bfd5d9f6-d9ft9" Jul 21 00:34:13.192397 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:13.192311 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d75f19e5-a207-4df5-a72e-9782ba3927a8-webhook-cert\") pod \"maas-controller-79bfd5d9f6-d9ft9\" (UID: \"d75f19e5-a207-4df5-a72e-9782ba3927a8\") " pod="opendatahub/maas-controller-79bfd5d9f6-d9ft9" Jul 21 00:34:13.192549 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:13.192397 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pxzt\" (UniqueName: \"kubernetes.io/projected/d75f19e5-a207-4df5-a72e-9782ba3927a8-kube-api-access-6pxzt\") pod \"maas-controller-79bfd5d9f6-d9ft9\" (UID: \"d75f19e5-a207-4df5-a72e-9782ba3927a8\") " pod="opendatahub/maas-controller-79bfd5d9f6-d9ft9" Jul 21 00:34:13.195212 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:13.195183 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d75f19e5-a207-4df5-a72e-9782ba3927a8-webhook-cert\") pod \"maas-controller-79bfd5d9f6-d9ft9\" (UID: \"d75f19e5-a207-4df5-a72e-9782ba3927a8\") " pod="opendatahub/maas-controller-79bfd5d9f6-d9ft9" Jul 21 00:34:13.201628 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:13.201558 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pxzt\" (UniqueName: \"kubernetes.io/projected/d75f19e5-a207-4df5-a72e-9782ba3927a8-kube-api-access-6pxzt\") pod \"maas-controller-79bfd5d9f6-d9ft9\" (UID: \"d75f19e5-a207-4df5-a72e-9782ba3927a8\") " pod="opendatahub/maas-controller-79bfd5d9f6-d9ft9" Jul 21 00:34:13.308823 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:13.308767 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-79bfd5d9f6-d9ft9" Jul 21 00:34:13.461643 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:13.461614 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-79bfd5d9f6-d9ft9"] Jul 21 00:34:13.463579 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:34:13.463550 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75f19e5_a207_4df5_a72e_9782ba3927a8.slice/crio-b84f207bf7997fb2f602c6fb7222496f96e3b8399a4df3a15065e20be3543f08 WatchSource:0}: Error finding container b84f207bf7997fb2f602c6fb7222496f96e3b8399a4df3a15065e20be3543f08: Status 404 returned error can't find the container with id b84f207bf7997fb2f602c6fb7222496f96e3b8399a4df3a15065e20be3543f08 Jul 21 00:34:14.204642 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:14.204517 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79bfd5d9f6-d9ft9" event={"ID":"d75f19e5-a207-4df5-a72e-9782ba3927a8","Type":"ContainerStarted","Data":"242f2dfbc7887764aee85aca8fd31bfe60c9343ea0d52eade81f845c53ac7374"} Jul 21 00:34:14.204642 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:14.204577 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-79bfd5d9f6-d9ft9" event={"ID":"d75f19e5-a207-4df5-a72e-9782ba3927a8","Type":"ContainerStarted","Data":"b84f207bf7997fb2f602c6fb7222496f96e3b8399a4df3a15065e20be3543f08"} Jul 21 00:34:14.205038 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:14.204731 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-79bfd5d9f6-d9ft9" Jul 21 00:34:14.223057 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:14.222989 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-79bfd5d9f6-d9ft9" podStartSLOduration=1.801052942 podStartE2EDuration="2.222966554s" totalImagesPullingTime="421.913612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:34:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:34:14.22205747 +0000 UTC m=+1048.437437796" watchObservedRunningTime="2026-07-21 00:34:14.222966554 +0000 UTC m=+1048.438346880" Jul 21 00:34:17.418126 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:17.418088 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:34:17.530700 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:17.530661 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:34:25.213134 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:25.213104 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-79bfd5d9f6-d9ft9" Jul 21 00:34:46.478831 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:46.478774 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:34:46.578867 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:46.578825 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:34:46.725270 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:46.725230 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:34:52.361983 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:52.361943 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:34:52.469513 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:52.469461 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:34:52.562509 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:52.562474 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:34:52.729033 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:52.728945 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:34:58.564307 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:58.564267 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:34:58.635085 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:34:58.635047 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:36:06.332631 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:36:06.332582 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:36:07.884358 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:36:07.884312 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:36:26.064771 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:36:26.064736 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:37:29.678446 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:37:29.678406 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:40:58.922286 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:40:58.922249 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:41:35.446171 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:41:35.446138 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:41:37.178846 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:41:37.178811 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:42:39.348785 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.348704 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn"] Jul 21 00:42:39.354767 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.354728 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:42:39.358409 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.358368 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-e58bdff0-kserve-self-signed-certs\"" Jul 21 00:42:39.371792 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.371759 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn"] Jul 21 00:42:39.377788 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.377751 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:42:39.440717 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.440675 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5be2404d-6b21-49be-9824-79c994de2bcd-tls-certs\") pod \"e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn\" (UID: \"5be2404d-6b21-49be-9824-79c994de2bcd\") " pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:42:39.440717 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.440723 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5be2404d-6b21-49be-9824-79c994de2bcd-home\") pod \"e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn\" (UID: \"5be2404d-6b21-49be-9824-79c994de2bcd\") " pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:42:39.440941 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.440799 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhq8p\" (UniqueName: \"kubernetes.io/projected/5be2404d-6b21-49be-9824-79c994de2bcd-kube-api-access-fhq8p\") pod \"e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn\" (UID: \"5be2404d-6b21-49be-9824-79c994de2bcd\") " pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:42:39.440941 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.440897 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5be2404d-6b21-49be-9824-79c994de2bcd-tmp-dir\") pod \"e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn\" (UID: \"5be2404d-6b21-49be-9824-79c994de2bcd\") " pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:42:39.440941 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.440929 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5be2404d-6b21-49be-9824-79c994de2bcd-model-cache\") pod \"e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn\" (UID: \"5be2404d-6b21-49be-9824-79c994de2bcd\") " pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:42:39.441038 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.440946 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5be2404d-6b21-49be-9824-79c994de2bcd-dshm\") pod \"e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn\" (UID: \"5be2404d-6b21-49be-9824-79c994de2bcd\") " pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:42:39.541738 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.541696 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5be2404d-6b21-49be-9824-79c994de2bcd-tmp-dir\") pod \"e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn\" (UID: \"5be2404d-6b21-49be-9824-79c994de2bcd\") " pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:42:39.541926 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.541752 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5be2404d-6b21-49be-9824-79c994de2bcd-model-cache\") pod \"e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn\" (UID: \"5be2404d-6b21-49be-9824-79c994de2bcd\") " pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:42:39.541926 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.541780 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5be2404d-6b21-49be-9824-79c994de2bcd-dshm\") pod \"e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn\" (UID: \"5be2404d-6b21-49be-9824-79c994de2bcd\") " pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:42:39.541926 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.541832 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5be2404d-6b21-49be-9824-79c994de2bcd-tls-certs\") pod \"e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn\" (UID: \"5be2404d-6b21-49be-9824-79c994de2bcd\") " pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:42:39.541926 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.541868 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5be2404d-6b21-49be-9824-79c994de2bcd-home\") pod \"e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn\" (UID: \"5be2404d-6b21-49be-9824-79c994de2bcd\") " pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:42:39.541926 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.541917 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhq8p\" (UniqueName: \"kubernetes.io/projected/5be2404d-6b21-49be-9824-79c994de2bcd-kube-api-access-fhq8p\") pod \"e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn\" (UID: \"5be2404d-6b21-49be-9824-79c994de2bcd\") " pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:42:39.542197 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.542173 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5be2404d-6b21-49be-9824-79c994de2bcd-tmp-dir\") pod \"e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn\" (UID: \"5be2404d-6b21-49be-9824-79c994de2bcd\") " pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:42:39.542259 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.542224 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5be2404d-6b21-49be-9824-79c994de2bcd-model-cache\") pod \"e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn\" (UID: \"5be2404d-6b21-49be-9824-79c994de2bcd\") " pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:42:39.542330 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.542308 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5be2404d-6b21-49be-9824-79c994de2bcd-home\") pod \"e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn\" (UID: \"5be2404d-6b21-49be-9824-79c994de2bcd\") " pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:42:39.544253 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.544221 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5be2404d-6b21-49be-9824-79c994de2bcd-dshm\") pod \"e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn\" (UID: \"5be2404d-6b21-49be-9824-79c994de2bcd\") " pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:42:39.544477 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.544455 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5be2404d-6b21-49be-9824-79c994de2bcd-tls-certs\") pod \"e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn\" (UID: \"5be2404d-6b21-49be-9824-79c994de2bcd\") " pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:42:39.550489 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.550462 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhq8p\" (UniqueName: \"kubernetes.io/projected/5be2404d-6b21-49be-9824-79c994de2bcd-kube-api-access-fhq8p\") pod \"e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn\" (UID: \"5be2404d-6b21-49be-9824-79c994de2bcd\") " pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:42:39.667582 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.667482 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:42:39.830580 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:39.830552 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn"] Jul 21 00:42:39.832207 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:42:39.832175 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5be2404d_6b21_49be_9824_79c994de2bcd.slice/crio-3be3fd4b11e08a76d158e316edc4acee33c4841a110f619e7e643002417415ad WatchSource:0}: Error finding container 3be3fd4b11e08a76d158e316edc4acee33c4841a110f619e7e643002417415ad: Status 404 returned error can't find the container with id 3be3fd4b11e08a76d158e316edc4acee33c4841a110f619e7e643002417415ad Jul 21 00:42:40.063108 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:40.063069 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" event={"ID":"5be2404d-6b21-49be-9824-79c994de2bcd","Type":"ContainerStarted","Data":"3497f713eb21d4bd6d32ef1bad7eb1e5268f0dab78127d114b95ee9469e9192c"} Jul 21 00:42:40.063108 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:40.063121 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" event={"ID":"5be2404d-6b21-49be-9824-79c994de2bcd","Type":"ContainerStarted","Data":"3be3fd4b11e08a76d158e316edc4acee33c4841a110f619e7e643002417415ad"} Jul 21 00:42:40.081836 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:40.081776 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" podStartSLOduration=1.081760254 podStartE2EDuration="1.081760254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:42:40.078923136 +0000 UTC m=+1554.294303462" watchObservedRunningTime="2026-07-21 00:42:40.081760254 +0000 UTC m=+1554.297140570" Jul 21 00:42:49.162388 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:49.162351 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:42:49.668621 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:49.668575 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:42:49.668839 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:49.668659 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:42:49.681148 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:49.681117 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:42:50.133023 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:42:50.132984 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:43:24.495399 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:24.495365 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn"] Jul 21 00:43:24.522168 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:24.522127 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:43:25.439831 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:25.439795 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:43:39.517088 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:39.516992 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" podUID="5be2404d-6b21-49be-9824-79c994de2bcd" containerName="main" containerID="cri-o://3497f713eb21d4bd6d32ef1bad7eb1e5268f0dab78127d114b95ee9469e9192c" gracePeriod=45 Jul 21 00:43:39.773940 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:39.773876 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:43:39.801161 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:39.801070 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5be2404d-6b21-49be-9824-79c994de2bcd-tls-certs\") pod \"5be2404d-6b21-49be-9824-79c994de2bcd\" (UID: \"5be2404d-6b21-49be-9824-79c994de2bcd\") " Jul 21 00:43:39.801345 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:39.801185 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5be2404d-6b21-49be-9824-79c994de2bcd-model-cache\") pod \"5be2404d-6b21-49be-9824-79c994de2bcd\" (UID: \"5be2404d-6b21-49be-9824-79c994de2bcd\") " Jul 21 00:43:39.801345 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:39.801205 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5be2404d-6b21-49be-9824-79c994de2bcd-dshm\") pod \"5be2404d-6b21-49be-9824-79c994de2bcd\" (UID: \"5be2404d-6b21-49be-9824-79c994de2bcd\") " Jul 21 00:43:39.801345 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:39.801225 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5be2404d-6b21-49be-9824-79c994de2bcd-home\") pod \"5be2404d-6b21-49be-9824-79c994de2bcd\" (UID: \"5be2404d-6b21-49be-9824-79c994de2bcd\") " Jul 21 00:43:39.801345 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:39.801297 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhq8p\" (UniqueName: \"kubernetes.io/projected/5be2404d-6b21-49be-9824-79c994de2bcd-kube-api-access-fhq8p\") pod \"5be2404d-6b21-49be-9824-79c994de2bcd\" (UID: \"5be2404d-6b21-49be-9824-79c994de2bcd\") " Jul 21 00:43:39.801345 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:39.801316 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5be2404d-6b21-49be-9824-79c994de2bcd-tmp-dir\") pod \"5be2404d-6b21-49be-9824-79c994de2bcd\" (UID: \"5be2404d-6b21-49be-9824-79c994de2bcd\") " Jul 21 00:43:39.801650 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:39.801435 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be2404d-6b21-49be-9824-79c994de2bcd-model-cache" (OuterVolumeSpecName: "model-cache") pod "5be2404d-6b21-49be-9824-79c994de2bcd" (UID: "5be2404d-6b21-49be-9824-79c994de2bcd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:43:39.801650 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:39.801594 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5be2404d-6b21-49be-9824-79c994de2bcd-model-cache\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:43:39.801650 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:39.801615 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be2404d-6b21-49be-9824-79c994de2bcd-home" (OuterVolumeSpecName: "home") pod "5be2404d-6b21-49be-9824-79c994de2bcd" (UID: "5be2404d-6b21-49be-9824-79c994de2bcd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:43:39.801814 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:39.801645 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be2404d-6b21-49be-9824-79c994de2bcd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5be2404d-6b21-49be-9824-79c994de2bcd" (UID: "5be2404d-6b21-49be-9824-79c994de2bcd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:43:39.803410 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:39.803384 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5be2404d-6b21-49be-9824-79c994de2bcd-dshm" (OuterVolumeSpecName: "dshm") pod "5be2404d-6b21-49be-9824-79c994de2bcd" (UID: "5be2404d-6b21-49be-9824-79c994de2bcd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:43:39.803533 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:39.803466 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5be2404d-6b21-49be-9824-79c994de2bcd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5be2404d-6b21-49be-9824-79c994de2bcd" (UID: "5be2404d-6b21-49be-9824-79c994de2bcd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:43:39.803533 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:39.803506 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be2404d-6b21-49be-9824-79c994de2bcd-kube-api-access-fhq8p" (OuterVolumeSpecName: "kube-api-access-fhq8p") pod "5be2404d-6b21-49be-9824-79c994de2bcd" (UID: "5be2404d-6b21-49be-9824-79c994de2bcd"). InnerVolumeSpecName "kube-api-access-fhq8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:43:39.902737 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:39.902707 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhq8p\" (UniqueName: \"kubernetes.io/projected/5be2404d-6b21-49be-9824-79c994de2bcd-kube-api-access-fhq8p\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:43:39.902908 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:39.902748 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5be2404d-6b21-49be-9824-79c994de2bcd-tmp-dir\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:43:39.902908 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:39.902759 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5be2404d-6b21-49be-9824-79c994de2bcd-tls-certs\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:43:39.902908 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:39.902767 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5be2404d-6b21-49be-9824-79c994de2bcd-dshm\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:43:39.902908 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:39.902775 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5be2404d-6b21-49be-9824-79c994de2bcd-home\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:43:40.408685 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:40.408652 2580 generic.go:356] "Generic (PLEG): container finished" podID="5be2404d-6b21-49be-9824-79c994de2bcd" containerID="3497f713eb21d4bd6d32ef1bad7eb1e5268f0dab78127d114b95ee9469e9192c" exitCode=0 Jul 21 00:43:40.408889 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:40.408709 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" Jul 21 00:43:40.408889 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:40.408719 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" event={"ID":"5be2404d-6b21-49be-9824-79c994de2bcd","Type":"ContainerDied","Data":"3497f713eb21d4bd6d32ef1bad7eb1e5268f0dab78127d114b95ee9469e9192c"} Jul 21 00:43:40.408889 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:40.408765 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn" event={"ID":"5be2404d-6b21-49be-9824-79c994de2bcd","Type":"ContainerDied","Data":"3be3fd4b11e08a76d158e316edc4acee33c4841a110f619e7e643002417415ad"} Jul 21 00:43:40.408889 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:40.408786 2580 scope.go:117] "RemoveContainer" containerID="3497f713eb21d4bd6d32ef1bad7eb1e5268f0dab78127d114b95ee9469e9192c" Jul 21 00:43:40.417884 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:40.417862 2580 scope.go:117] "RemoveContainer" containerID="3497f713eb21d4bd6d32ef1bad7eb1e5268f0dab78127d114b95ee9469e9192c" Jul 21 00:43:40.418216 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:43:40.418193 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3497f713eb21d4bd6d32ef1bad7eb1e5268f0dab78127d114b95ee9469e9192c\": container with ID starting with 3497f713eb21d4bd6d32ef1bad7eb1e5268f0dab78127d114b95ee9469e9192c not found: ID does not exist" containerID="3497f713eb21d4bd6d32ef1bad7eb1e5268f0dab78127d114b95ee9469e9192c" Jul 21 00:43:40.418324 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:40.418223 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3497f713eb21d4bd6d32ef1bad7eb1e5268f0dab78127d114b95ee9469e9192c"} err="failed to get container status \"3497f713eb21d4bd6d32ef1bad7eb1e5268f0dab78127d114b95ee9469e9192c\": rpc error: code = NotFound desc = could not find container \"3497f713eb21d4bd6d32ef1bad7eb1e5268f0dab78127d114b95ee9469e9192c\": container with ID starting with 3497f713eb21d4bd6d32ef1bad7eb1e5268f0dab78127d114b95ee9469e9192c not found: ID does not exist" Jul 21 00:43:40.444453 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:40.444417 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn"] Jul 21 00:43:40.461267 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:40.461235 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-e58bdff0-kserve-796564cf-qwknn"] Jul 21 00:43:42.297722 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:43:42.297686 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5be2404d-6b21-49be-9824-79c994de2bcd" path="/var/lib/kubelet/pods/5be2404d-6b21-49be-9824-79c994de2bcd/volumes" Jul 21 00:45:00.139445 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:45:00.139412 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29743245-sjvv4"] Jul 21 00:45:00.139961 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:45:00.139845 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5be2404d-6b21-49be-9824-79c994de2bcd" containerName="main" Jul 21 00:45:00.139961 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:45:00.139856 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5be2404d-6b21-49be-9824-79c994de2bcd" containerName="main" Jul 21 00:45:00.139961 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:45:00.139913 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5be2404d-6b21-49be-9824-79c994de2bcd" containerName="main" Jul 21 00:45:00.142981 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:45:00.142964 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743245-sjvv4" Jul 21 00:45:00.145375 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:45:00.145346 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxv7p\"" Jul 21 00:45:00.149460 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:45:00.149428 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29743245-sjvv4"] Jul 21 00:45:00.154378 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:45:00.154355 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743245-sjvv4" Jul 21 00:45:00.503238 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:45:00.503198 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29743245-sjvv4"] Jul 21 00:45:00.504472 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:45:00.504441 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd4ac73_6cd5_4f40_964c_c584155856c8.slice/crio-a0a383fae6a71f6edca6972cf1d358be921c187bc67e2d59a05e6c5eca82b85b WatchSource:0}: Error finding container a0a383fae6a71f6edca6972cf1d358be921c187bc67e2d59a05e6c5eca82b85b: Status 404 returned error can't find the container with id a0a383fae6a71f6edca6972cf1d358be921c187bc67e2d59a05e6c5eca82b85b Jul 21 00:45:00.872567 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:45:00.872531 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743245-sjvv4" event={"ID":"fbd4ac73-6cd5-4f40-964c-c584155856c8","Type":"ContainerStarted","Data":"011d37fc0b96578118978ef324914d7f9efb42e4076c672350d9981ea0e061d6"} Jul 21 00:45:00.872764 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:45:00.872585 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743245-sjvv4" event={"ID":"fbd4ac73-6cd5-4f40-964c-c584155856c8","Type":"ContainerStarted","Data":"a0a383fae6a71f6edca6972cf1d358be921c187bc67e2d59a05e6c5eca82b85b"} Jul 21 00:45:00.887792 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:45:00.887734 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743245-sjvv4" podStartSLOduration=0.887717083 podStartE2EDuration="887.717083ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:45:00.885204141 +0000 UTC m=+1695.100584465" watchObservedRunningTime="2026-07-21 00:45:00.887717083 +0000 UTC m=+1695.103097408" Jul 21 00:45:01.879046 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:45:01.879011 2580 generic.go:356] "Generic (PLEG): container finished" podID="fbd4ac73-6cd5-4f40-964c-c584155856c8" containerID="011d37fc0b96578118978ef324914d7f9efb42e4076c672350d9981ea0e061d6" exitCode=0 Jul 21 00:45:01.879448 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:45:01.879098 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743245-sjvv4" event={"ID":"fbd4ac73-6cd5-4f40-964c-c584155856c8","Type":"ContainerDied","Data":"011d37fc0b96578118978ef324914d7f9efb42e4076c672350d9981ea0e061d6"} Jul 21 00:45:03.033378 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:45:03.033347 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743245-sjvv4" Jul 21 00:45:03.892682 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:45:03.892644 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743245-sjvv4" event={"ID":"fbd4ac73-6cd5-4f40-964c-c584155856c8","Type":"ContainerDied","Data":"a0a383fae6a71f6edca6972cf1d358be921c187bc67e2d59a05e6c5eca82b85b"} Jul 21 00:45:03.892870 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:45:03.892692 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29743245-sjvv4" Jul 21 00:45:03.892870 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:45:03.892695 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0a383fae6a71f6edca6972cf1d358be921c187bc67e2d59a05e6c5eca82b85b" Jul 21 00:46:32.839226 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:46:32.839191 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:46:40.571475 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:46:40.571393 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:46:48.857573 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:46:48.857533 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:46:54.157176 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:46:54.157131 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:46:57.315415 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:46:57.315350 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:46:57.414434 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:46:57.414396 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:46:57.783307 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:46:57.783224 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:03.488330 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:03.488290 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:03.561057 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:03.561019 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:03.729917 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:03.729884 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:04.006395 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.006354 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg"] Jul 21 00:47:04.006796 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.006783 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbd4ac73-6cd5-4f40-964c-c584155856c8" containerName="cleanup" Jul 21 00:47:04.006858 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.006797 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbd4ac73-6cd5-4f40-964c-c584155856c8" containerName="cleanup" Jul 21 00:47:04.006858 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.006856 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbd4ac73-6cd5-4f40-964c-c584155856c8" containerName="cleanup" Jul 21 00:47:04.010118 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.010094 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.012701 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.012684 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-865fdc2c-openshift-default-dockercfg-vpdwf\"" Jul 21 00:47:04.014287 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.014242 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istio-envoy\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.014381 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.014305 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istiod-ca-cert\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.014381 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.014361 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-workload-certs\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.014464 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.014382 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-credential-socket\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.014504 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.014463 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istio-token\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.014504 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.014492 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-workload-socket\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.014577 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.014566 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdmwx\" (UniqueName: \"kubernetes.io/projected/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-kube-api-access-qdmwx\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.014685 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.014667 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istio-data\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.014730 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.014717 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istio-podinfo\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.028905 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.028872 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg"] Jul 21 00:47:04.041400 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.041363 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:04.115190 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.115151 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-workload-certs\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.115190 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.115197 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-credential-socket\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.115463 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.115232 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istio-token\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.115463 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.115257 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-workload-socket\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.115463 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.115284 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdmwx\" (UniqueName: \"kubernetes.io/projected/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-kube-api-access-qdmwx\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.115463 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.115330 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istio-data\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.115463 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.115378 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istio-podinfo\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.115463 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.115407 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istio-envoy\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.115463 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.115433 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istiod-ca-cert\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.116029 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.115759 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-workload-certs\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.116029 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.115793 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-workload-socket\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.116029 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.116012 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-credential-socket\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.116736 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.116138 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istio-data\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.116736 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.116723 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istiod-ca-cert\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.118511 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.118484 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istio-podinfo\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.118885 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.118858 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istio-envoy\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.124147 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.124118 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istio-token\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.124273 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.124216 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdmwx\" (UniqueName: \"kubernetes.io/projected/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-kube-api-access-qdmwx\") pod \"e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.247046 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.247005 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:04.320729 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.320697 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:04.469937 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.469905 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg"] Jul 21 00:47:04.470947 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:47:04.470920 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c2e0d4_221c_4da1_8705_6b0ec4fd58c7.slice/crio-55fac8a93ae35551b88244372346e172bcf2d20b4bf7d306d8db62c6dc2b1a14 WatchSource:0}: Error finding container 55fac8a93ae35551b88244372346e172bcf2d20b4bf7d306d8db62c6dc2b1a14: Status 404 returned error can't find the container with id 55fac8a93ae35551b88244372346e172bcf2d20b4bf7d306d8db62c6dc2b1a14 Jul 21 00:47:04.473181 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.473142 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:47:04.473265 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.473247 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:47:04.473315 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.473299 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:47:04.582994 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.582882 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" event={"ID":"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7","Type":"ContainerStarted","Data":"ca68c62f447b5fc27f823ac9eab27db92d24f8c47b944d6dc60ab0692068917c"} Jul 21 00:47:04.582994 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.582947 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" event={"ID":"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7","Type":"ContainerStarted","Data":"55fac8a93ae35551b88244372346e172bcf2d20b4bf7d306d8db62c6dc2b1a14"} Jul 21 00:47:04.604509 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:04.604438 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" podStartSLOduration=1.604420092 podStartE2EDuration="1.604420092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:47:04.601634126 +0000 UTC m=+1818.817014450" watchObservedRunningTime="2026-07-21 00:47:04.604420092 +0000 UTC m=+1818.819800416" Jul 21 00:47:05.321229 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:05.321190 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:05.326656 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:05.326629 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:05.592700 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:05.592591 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:05.593832 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:05.593810 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:05.943319 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:05.943228 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:06.044635 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:06.044560 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:06.114812 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:06.114771 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:06.212941 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:06.212861 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:06.604429 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:06.604397 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:06.711673 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:06.711636 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:11.939765 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:11.939727 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:17.663707 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:17.663669 2580 generic.go:356] "Generic (PLEG): container finished" podID="96c2e0d4-221c-4da1-8705-6b0ec4fd58c7" containerID="ca68c62f447b5fc27f823ac9eab27db92d24f8c47b944d6dc60ab0692068917c" exitCode=0 Jul 21 00:47:17.664164 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:17.663744 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" event={"ID":"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7","Type":"ContainerDied","Data":"ca68c62f447b5fc27f823ac9eab27db92d24f8c47b944d6dc60ab0692068917c"} Jul 21 00:47:17.664216 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:17.664160 2580 scope.go:117] "RemoveContainer" containerID="ca68c62f447b5fc27f823ac9eab27db92d24f8c47b944d6dc60ab0692068917c" Jul 21 00:47:17.665187 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:17.665159 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:47:17.665250 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:17.665229 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:47:17.665294 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:17.665263 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:47:18.671982 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:18.671948 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" event={"ID":"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7","Type":"ContainerStarted","Data":"7f89e866fdc29f2e039da20f752107798fcf158d19096a070f68507bcd0a040f"} Jul 21 00:47:19.321428 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:19.321382 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:19.322915 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:19.322885 2580 patch_prober.go:28] interesting pod/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 21 00:47:19.323042 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:19.322968 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" podUID="96c2e0d4-221c-4da1-8705-6b0ec4fd58c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 21 00:47:20.321295 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:20.321252 2580 patch_prober.go:28] interesting pod/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 21 00:47:20.321770 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:20.321336 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" podUID="96c2e0d4-221c-4da1-8705-6b0ec4fd58c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 21 00:47:20.593717 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:20.593581 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:21.322046 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:21.322008 2580 patch_prober.go:28] interesting pod/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 21 00:47:21.322470 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:21.322081 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" podUID="96c2e0d4-221c-4da1-8705-6b0ec4fd58c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 21 00:47:22.322130 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:22.322086 2580 patch_prober.go:28] interesting pod/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 21 00:47:22.322538 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:22.322162 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" podUID="96c2e0d4-221c-4da1-8705-6b0ec4fd58c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 21 00:47:23.325675 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:23.325639 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:23.326662 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:23.326645 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:31.593657 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:31.593617 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865fdc2c-k7rn7"] Jul 21 00:47:31.597297 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:31.597273 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865fdc2c-k7rn7" Jul 21 00:47:31.600381 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:31.600358 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxv7p\"" Jul 21 00:47:31.603534 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:31.603504 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865fdc2c-k7rn7"] Jul 21 00:47:31.691418 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:31.691370 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa6995fa-83ae-45d8-a10a-1cc2d5e8141d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-865fdc2c-k7rn7\" (UID: \"fa6995fa-83ae-45d8-a10a-1cc2d5e8141d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865fdc2c-k7rn7" Jul 21 00:47:31.792917 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:31.792879 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa6995fa-83ae-45d8-a10a-1cc2d5e8141d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-865fdc2c-k7rn7\" (UID: \"fa6995fa-83ae-45d8-a10a-1cc2d5e8141d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865fdc2c-k7rn7" Jul 21 00:47:31.793525 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:31.793505 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa6995fa-83ae-45d8-a10a-1cc2d5e8141d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-865fdc2c-k7rn7\" (UID: \"fa6995fa-83ae-45d8-a10a-1cc2d5e8141d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865fdc2c-k7rn7" Jul 21 00:47:31.911927 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:31.911829 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865fdc2c-k7rn7" Jul 21 00:47:32.054874 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:32.054846 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865fdc2c-k7rn7"] Jul 21 00:47:32.056905 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:47:32.056872 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6995fa_83ae_45d8_a10a_1cc2d5e8141d.slice/crio-5bb0fc6c511838016742a26f44a263339062fc3c9c72f5f68a7f0d717f9e89e0 WatchSource:0}: Error finding container 5bb0fc6c511838016742a26f44a263339062fc3c9c72f5f68a7f0d717f9e89e0: Status 404 returned error can't find the container with id 5bb0fc6c511838016742a26f44a263339062fc3c9c72f5f68a7f0d717f9e89e0 Jul 21 00:47:32.760935 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:32.760834 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865fdc2c-k7rn7" event={"ID":"fa6995fa-83ae-45d8-a10a-1cc2d5e8141d","Type":"ContainerStarted","Data":"dee3f07649220b0d82c8da113500270e1cfb6a9dec99131561d9b778cb3bf8a9"} Jul 21 00:47:32.760935 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:32.760896 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865fdc2c-k7rn7" event={"ID":"fa6995fa-83ae-45d8-a10a-1cc2d5e8141d","Type":"ContainerStarted","Data":"5bb0fc6c511838016742a26f44a263339062fc3c9c72f5f68a7f0d717f9e89e0"} Jul 21 00:47:32.775132 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:32.775069 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865fdc2c-k7rn7" podStartSLOduration=1.7750532209999998 podStartE2EDuration="1.775053221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:47:32.774877233 +0000 UTC m=+1846.990257559" watchObservedRunningTime="2026-07-21 00:47:32.775053221 +0000 UTC m=+1846.990433547" Jul 21 00:47:33.767844 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:33.767804 2580 generic.go:356] "Generic (PLEG): container finished" podID="fa6995fa-83ae-45d8-a10a-1cc2d5e8141d" containerID="dee3f07649220b0d82c8da113500270e1cfb6a9dec99131561d9b778cb3bf8a9" exitCode=0 Jul 21 00:47:33.768266 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:33.767891 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865fdc2c-k7rn7" event={"ID":"fa6995fa-83ae-45d8-a10a-1cc2d5e8141d","Type":"ContainerDied","Data":"dee3f07649220b0d82c8da113500270e1cfb6a9dec99131561d9b778cb3bf8a9"} Jul 21 00:47:34.910049 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:34.910024 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865fdc2c-k7rn7" Jul 21 00:47:35.025326 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:35.025223 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa6995fa-83ae-45d8-a10a-1cc2d5e8141d-maas-api-service-ca\") pod \"fa6995fa-83ae-45d8-a10a-1cc2d5e8141d\" (UID: \"fa6995fa-83ae-45d8-a10a-1cc2d5e8141d\") " Jul 21 00:47:35.025590 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:35.025568 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa6995fa-83ae-45d8-a10a-1cc2d5e8141d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fa6995fa-83ae-45d8-a10a-1cc2d5e8141d" (UID: "fa6995fa-83ae-45d8-a10a-1cc2d5e8141d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:47:35.126023 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:35.125977 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa6995fa-83ae-45d8-a10a-1cc2d5e8141d-maas-api-service-ca\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:47:35.781677 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:35.781645 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865fdc2c-k7rn7" Jul 21 00:47:35.781677 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:35.781662 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865fdc2c-k7rn7" event={"ID":"fa6995fa-83ae-45d8-a10a-1cc2d5e8141d","Type":"ContainerDied","Data":"5bb0fc6c511838016742a26f44a263339062fc3c9c72f5f68a7f0d717f9e89e0"} Jul 21 00:47:35.781901 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:35.781717 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bb0fc6c511838016742a26f44a263339062fc3c9c72f5f68a7f0d717f9e89e0" Jul 21 00:47:42.171318 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:42.171286 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:46.676409 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:46.676373 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:46.678816 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:46.678790 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865fdc2c-k7rn7"] Jul 21 00:47:46.681388 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:46.681366 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-865fdc2c-k7rn7"] Jul 21 00:47:46.830661 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:46.830626 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg"] Jul 21 00:47:46.830904 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:46.830850 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" podUID="96c2e0d4-221c-4da1-8705-6b0ec4fd58c7" containerName="istio-proxy" containerID="cri-o://7f89e866fdc29f2e039da20f752107798fcf158d19096a070f68507bcd0a040f" gracePeriod=30 Jul 21 00:47:46.852004 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:46.851961 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:47.104736 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:47.104696 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:47.368086 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:47.367965 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:47.470728 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:47.470695 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:47.598045 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:47.598009 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:47.884044 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:47.884014 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch"] Jul 21 00:47:47.884468 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:47.884454 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa6995fa-83ae-45d8-a10a-1cc2d5e8141d" containerName="revoke-keys" Jul 21 00:47:47.884511 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:47.884469 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa6995fa-83ae-45d8-a10a-1cc2d5e8141d" containerName="revoke-keys" Jul 21 00:47:47.884545 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:47.884528 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa6995fa-83ae-45d8-a10a-1cc2d5e8141d" containerName="revoke-keys" Jul 21 00:47:47.887727 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:47.887707 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:47.891243 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:47.891217 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-c7cbed04-gw-openshift-default-dockercfg-zpx8r\"" Jul 21 00:47:47.901708 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:47.901680 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch"] Jul 21 00:47:47.922877 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:47.922850 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:47.936406 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:47.936352 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/174d0146-7c29-44ab-ae3d-579b701131d8-istiod-ca-cert\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:47.936406 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:47.936408 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-workload-certs\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:47.936689 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:47.936472 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/174d0146-7c29-44ab-ae3d-579b701131d8-istio-podinfo\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:47.936689 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:47.936560 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-istio-data\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:47.936689 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:47.936613 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-credential-socket\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:47.936689 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:47.936656 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v8vv\" (UniqueName: \"kubernetes.io/projected/174d0146-7c29-44ab-ae3d-579b701131d8-kube-api-access-2v8vv\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:47.936689 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:47.936689 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-workload-socket\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:47.936927 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:47.936712 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-istio-envoy\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:47.936927 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:47.936772 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/174d0146-7c29-44ab-ae3d-579b701131d8-istio-token\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:48.037901 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.037862 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/174d0146-7c29-44ab-ae3d-579b701131d8-istiod-ca-cert\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:48.038094 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.037918 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-workload-certs\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:48.038094 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.037970 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/174d0146-7c29-44ab-ae3d-579b701131d8-istio-podinfo\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:48.038094 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.038027 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-istio-data\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:48.038094 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.038055 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-credential-socket\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:48.038094 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.038090 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v8vv\" (UniqueName: \"kubernetes.io/projected/174d0146-7c29-44ab-ae3d-579b701131d8-kube-api-access-2v8vv\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:48.038336 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.038139 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-workload-socket\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:48.038336 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.038178 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-istio-envoy\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:48.038336 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.038227 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/174d0146-7c29-44ab-ae3d-579b701131d8-istio-token\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:48.038520 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.038346 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-workload-certs\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:48.038520 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.038423 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-istio-data\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:48.038652 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.038575 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-credential-socket\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:48.038652 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.038633 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-workload-socket\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:48.038900 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.038863 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/174d0146-7c29-44ab-ae3d-579b701131d8-istiod-ca-cert\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:48.044149 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.041908 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/174d0146-7c29-44ab-ae3d-579b701131d8-istio-podinfo\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:48.044149 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.042801 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-istio-envoy\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:48.050104 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.050070 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/174d0146-7c29-44ab-ae3d-579b701131d8-istio-token\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:48.050328 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.050313 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v8vv\" (UniqueName: \"kubernetes.io/projected/174d0146-7c29-44ab-ae3d-579b701131d8-kube-api-access-2v8vv\") pod \"e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:48.093843 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.093802 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:48.199768 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.199675 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:48.300268 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.300233 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa6995fa-83ae-45d8-a10a-1cc2d5e8141d" path="/var/lib/kubelet/pods/fa6995fa-83ae-45d8-a10a-1cc2d5e8141d/volumes" Jul 21 00:47:48.366999 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.366909 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch"] Jul 21 00:47:48.369903 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:47:48.369874 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174d0146_7c29_44ab_ae3d_579b701131d8.slice/crio-fbcf56aa72bccd279cfb0a0ff3eca2cd56d8a933639ad0426304f270734ead54 WatchSource:0}: Error finding container fbcf56aa72bccd279cfb0a0ff3eca2cd56d8a933639ad0426304f270734ead54: Status 404 returned error can't find the container with id fbcf56aa72bccd279cfb0a0ff3eca2cd56d8a933639ad0426304f270734ead54 Jul 21 00:47:48.371950 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.371918 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:47:48.372041 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.371984 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:47:48.372041 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.372012 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:47:48.869847 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.869807 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" event={"ID":"174d0146-7c29-44ab-ae3d-579b701131d8","Type":"ContainerStarted","Data":"dde8179fb7abde097c13372cd329279e5550cc4342ef15afcfaefe5a92d28ae7"} Jul 21 00:47:48.870065 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.869874 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" event={"ID":"174d0146-7c29-44ab-ae3d-579b701131d8","Type":"ContainerStarted","Data":"fbcf56aa72bccd279cfb0a0ff3eca2cd56d8a933639ad0426304f270734ead54"} Jul 21 00:47:48.899910 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:48.899852 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" podStartSLOduration=1.899835094 podStartE2EDuration="1.899835094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:47:48.896725 +0000 UTC m=+1863.112105325" watchObservedRunningTime="2026-07-21 00:47:48.899835094 +0000 UTC m=+1863.115215475" Jul 21 00:47:49.200138 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:49.200049 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:49.205151 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:49.205125 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:49.876130 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:49.876101 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:49.877225 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:49.877204 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:47:50.426380 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:50.426346 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:50.747250 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:50.747164 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:50.989120 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:50.989086 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:51.315672 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:51.315635 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:51.419279 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:51.419238 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:47:52.085437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.085413 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:52.180531 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.180441 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istiod-ca-cert\") pod \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " Jul 21 00:47:52.180531 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.180479 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-workload-certs\") pod \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " Jul 21 00:47:52.180531 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.180524 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-workload-socket\") pod \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " Jul 21 00:47:52.180843 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.180552 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-credential-socket\") pod \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " Jul 21 00:47:52.180843 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.180586 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdmwx\" (UniqueName: \"kubernetes.io/projected/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-kube-api-access-qdmwx\") pod \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " Jul 21 00:47:52.180843 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.180641 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istio-data\") pod \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " Jul 21 00:47:52.180843 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.180689 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istio-podinfo\") pod \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " Jul 21 00:47:52.180843 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.180736 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istio-envoy\") pod \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " Jul 21 00:47:52.180843 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.180799 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istio-token\") pod \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\" (UID: \"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7\") " Jul 21 00:47:52.181162 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.180879 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "96c2e0d4-221c-4da1-8705-6b0ec4fd58c7" (UID: "96c2e0d4-221c-4da1-8705-6b0ec4fd58c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:47:52.181162 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.180889 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "96c2e0d4-221c-4da1-8705-6b0ec4fd58c7" (UID: "96c2e0d4-221c-4da1-8705-6b0ec4fd58c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:47:52.181162 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.181150 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-workload-certs\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:47:52.181313 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.181172 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-workload-socket\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:47:52.181313 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.181173 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "96c2e0d4-221c-4da1-8705-6b0ec4fd58c7" (UID: "96c2e0d4-221c-4da1-8705-6b0ec4fd58c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:47:52.181313 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.181223 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "96c2e0d4-221c-4da1-8705-6b0ec4fd58c7" (UID: "96c2e0d4-221c-4da1-8705-6b0ec4fd58c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:47:52.181680 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.181652 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istio-data" (OuterVolumeSpecName: "istio-data") pod "96c2e0d4-221c-4da1-8705-6b0ec4fd58c7" (UID: "96c2e0d4-221c-4da1-8705-6b0ec4fd58c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:47:52.183708 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.183666 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "96c2e0d4-221c-4da1-8705-6b0ec4fd58c7" (UID: "96c2e0d4-221c-4da1-8705-6b0ec4fd58c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:47:52.183708 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.183675 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istio-token" (OuterVolumeSpecName: "istio-token") pod "96c2e0d4-221c-4da1-8705-6b0ec4fd58c7" (UID: "96c2e0d4-221c-4da1-8705-6b0ec4fd58c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:47:52.183861 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.183731 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "96c2e0d4-221c-4da1-8705-6b0ec4fd58c7" (UID: "96c2e0d4-221c-4da1-8705-6b0ec4fd58c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 00:47:52.183861 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.183768 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-kube-api-access-qdmwx" (OuterVolumeSpecName: "kube-api-access-qdmwx") pod "96c2e0d4-221c-4da1-8705-6b0ec4fd58c7" (UID: "96c2e0d4-221c-4da1-8705-6b0ec4fd58c7"). InnerVolumeSpecName "kube-api-access-qdmwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:47:52.282563 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.282523 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istiod-ca-cert\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:47:52.282563 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.282558 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-credential-socket\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:47:52.282563 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.282569 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdmwx\" (UniqueName: \"kubernetes.io/projected/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-kube-api-access-qdmwx\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:47:52.282823 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.282579 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istio-data\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:47:52.282823 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.282589 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istio-podinfo\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:47:52.282823 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.282627 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istio-envoy\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:47:52.282823 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.282639 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7-istio-token\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:47:52.894917 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.894884 2580 generic.go:356] "Generic (PLEG): container finished" podID="96c2e0d4-221c-4da1-8705-6b0ec4fd58c7" containerID="7f89e866fdc29f2e039da20f752107798fcf158d19096a070f68507bcd0a040f" exitCode=0 Jul 21 00:47:52.895112 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.894939 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" Jul 21 00:47:52.895112 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.894976 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" event={"ID":"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7","Type":"ContainerDied","Data":"7f89e866fdc29f2e039da20f752107798fcf158d19096a070f68507bcd0a040f"} Jul 21 00:47:52.895112 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.895032 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg" event={"ID":"96c2e0d4-221c-4da1-8705-6b0ec4fd58c7","Type":"ContainerDied","Data":"55fac8a93ae35551b88244372346e172bcf2d20b4bf7d306d8db62c6dc2b1a14"} Jul 21 00:47:52.895112 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.895055 2580 scope.go:117] "RemoveContainer" containerID="7f89e866fdc29f2e039da20f752107798fcf158d19096a070f68507bcd0a040f" Jul 21 00:47:52.905165 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.905144 2580 scope.go:117] "RemoveContainer" containerID="ca68c62f447b5fc27f823ac9eab27db92d24f8c47b944d6dc60ab0692068917c" Jul 21 00:47:52.913214 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.913185 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg"] Jul 21 00:47:52.915659 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.915637 2580 scope.go:117] "RemoveContainer" containerID="7f89e866fdc29f2e039da20f752107798fcf158d19096a070f68507bcd0a040f" Jul 21 00:47:52.915988 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:47:52.915961 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f89e866fdc29f2e039da20f752107798fcf158d19096a070f68507bcd0a040f\": container with ID starting with 7f89e866fdc29f2e039da20f752107798fcf158d19096a070f68507bcd0a040f not found: ID does not exist" containerID="7f89e866fdc29f2e039da20f752107798fcf158d19096a070f68507bcd0a040f" Jul 21 00:47:52.916087 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.916002 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f89e866fdc29f2e039da20f752107798fcf158d19096a070f68507bcd0a040f"} err="failed to get container status \"7f89e866fdc29f2e039da20f752107798fcf158d19096a070f68507bcd0a040f\": rpc error: code = NotFound desc = could not find container \"7f89e866fdc29f2e039da20f752107798fcf158d19096a070f68507bcd0a040f\": container with ID starting with 7f89e866fdc29f2e039da20f752107798fcf158d19096a070f68507bcd0a040f not found: ID does not exist" Jul 21 00:47:52.916087 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.916027 2580 scope.go:117] "RemoveContainer" containerID="ca68c62f447b5fc27f823ac9eab27db92d24f8c47b944d6dc60ab0692068917c" Jul 21 00:47:52.916265 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:47:52.916248 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca68c62f447b5fc27f823ac9eab27db92d24f8c47b944d6dc60ab0692068917c\": container with ID starting with ca68c62f447b5fc27f823ac9eab27db92d24f8c47b944d6dc60ab0692068917c not found: ID does not exist" containerID="ca68c62f447b5fc27f823ac9eab27db92d24f8c47b944d6dc60ab0692068917c" Jul 21 00:47:52.916303 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.916274 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca68c62f447b5fc27f823ac9eab27db92d24f8c47b944d6dc60ab0692068917c"} err="failed to get container status \"ca68c62f447b5fc27f823ac9eab27db92d24f8c47b944d6dc60ab0692068917c\": rpc error: code = NotFound desc = could not find container \"ca68c62f447b5fc27f823ac9eab27db92d24f8c47b944d6dc60ab0692068917c\": container with ID starting with ca68c62f447b5fc27f823ac9eab27db92d24f8c47b944d6dc60ab0692068917c not found: ID does not exist" Jul 21 00:47:52.917086 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:52.917066 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-865fdc2c-openshift-default-77685bb6c7-x46wg"] Jul 21 00:47:54.298143 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:47:54.298106 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96c2e0d4-221c-4da1-8705-6b0ec4fd58c7" path="/var/lib/kubelet/pods/96c2e0d4-221c-4da1-8705-6b0ec4fd58c7/volumes" Jul 21 00:48:04.877888 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:04.877849 2580 patch_prober.go:28] interesting pod/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 21 00:48:04.878305 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:04.877913 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" podUID="174d0146-7c29-44ab-ae3d-579b701131d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 21 00:48:04.973028 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:04.972994 2580 generic.go:356] "Generic (PLEG): container finished" podID="174d0146-7c29-44ab-ae3d-579b701131d8" containerID="dde8179fb7abde097c13372cd329279e5550cc4342ef15afcfaefe5a92d28ae7" exitCode=0 Jul 21 00:48:04.973204 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:04.973068 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" event={"ID":"174d0146-7c29-44ab-ae3d-579b701131d8","Type":"ContainerDied","Data":"dde8179fb7abde097c13372cd329279e5550cc4342ef15afcfaefe5a92d28ae7"} Jul 21 00:48:04.973491 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:04.973477 2580 scope.go:117] "RemoveContainer" containerID="dde8179fb7abde097c13372cd329279e5550cc4342ef15afcfaefe5a92d28ae7" Jul 21 00:48:04.974484 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:04.974455 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:48:04.974551 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:04.974526 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:48:04.974613 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:04.974556 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:48:05.982033 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:05.981997 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" event={"ID":"174d0146-7c29-44ab-ae3d-579b701131d8","Type":"ContainerStarted","Data":"892441250fe59d6b73e4f4fc94d519e3551d1fce4a9e4039950039d47c84ecc0"} Jul 21 00:48:06.200773 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:06.200728 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:48:06.201002 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:06.200797 2580 patch_prober.go:28] interesting pod/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 21 00:48:06.201002 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:06.200864 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" podUID="174d0146-7c29-44ab-ae3d-579b701131d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 21 00:48:07.200400 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:07.200283 2580 patch_prober.go:28] interesting pod/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 21 00:48:07.200400 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:07.200363 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" podUID="174d0146-7c29-44ab-ae3d-579b701131d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 21 00:48:08.200346 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:08.200293 2580 patch_prober.go:28] interesting pod/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 21 00:48:08.200615 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:08.200373 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" podUID="174d0146-7c29-44ab-ae3d-579b701131d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 21 00:48:09.200367 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:09.200325 2580 patch_prober.go:28] interesting pod/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 21 00:48:09.200715 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:09.200684 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" podUID="174d0146-7c29-44ab-ae3d-579b701131d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 21 00:48:10.204768 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:10.204727 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:48:10.205245 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:10.205126 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:48:10.205759 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:10.205740 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:48:26.258282 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:26.258248 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:48:30.547764 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:30.547714 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:48:30.684670 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:30.684632 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch"] Jul 21 00:48:30.684937 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:30.684909 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" podUID="174d0146-7c29-44ab-ae3d-579b701131d8" containerName="istio-proxy" containerID="cri-o://892441250fe59d6b73e4f4fc94d519e3551d1fce4a9e4039950039d47c84ecc0" gracePeriod=30 Jul 21 00:48:30.708008 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:30.707959 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:48:30.965877 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:30.965837 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:48:31.235531 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.235442 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:48:31.523872 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.523776 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h"] Jul 21 00:48:31.524385 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.524365 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96c2e0d4-221c-4da1-8705-6b0ec4fd58c7" containerName="istio-proxy" Jul 21 00:48:31.524433 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.524389 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="96c2e0d4-221c-4da1-8705-6b0ec4fd58c7" containerName="istio-proxy" Jul 21 00:48:31.524433 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.524418 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96c2e0d4-221c-4da1-8705-6b0ec4fd58c7" containerName="istio-proxy" Jul 21 00:48:31.524433 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.524426 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="96c2e0d4-221c-4da1-8705-6b0ec4fd58c7" containerName="istio-proxy" Jul 21 00:48:31.524529 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.524519 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="96c2e0d4-221c-4da1-8705-6b0ec4fd58c7" containerName="istio-proxy" Jul 21 00:48:31.524574 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.524543 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="96c2e0d4-221c-4da1-8705-6b0ec4fd58c7" containerName="istio-proxy" Jul 21 00:48:31.528025 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.528007 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.531099 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.530958 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e23578bb-openshift-default-dockercfg-bg2ss\"" Jul 21 00:48:31.535075 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.534995 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h"] Jul 21 00:48:31.542779 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.542739 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:48:31.565223 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.565179 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-istio-envoy\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.565223 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.565221 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lht4b\" (UniqueName: \"kubernetes.io/projected/a4215607-32c3-4bf4-8063-a6bf1be43877-kube-api-access-lht4b\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.565758 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.565251 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4215607-32c3-4bf4-8063-a6bf1be43877-istio-token\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.565758 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.565394 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-istio-data\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.565758 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.565441 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4215607-32c3-4bf4-8063-a6bf1be43877-istio-podinfo\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.565758 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.565500 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4215607-32c3-4bf4-8063-a6bf1be43877-istiod-ca-cert\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.565758 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.565551 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-workload-certs\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.565758 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.565578 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-workload-socket\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.565758 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.565641 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-credential-socket\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.666383 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.666341 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-istio-data\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.666383 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.666383 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4215607-32c3-4bf4-8063-a6bf1be43877-istio-podinfo\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.666737 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.666404 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4215607-32c3-4bf4-8063-a6bf1be43877-istiod-ca-cert\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.666737 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.666435 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-workload-certs\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.666737 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.666454 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-workload-socket\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.666737 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.666494 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-credential-socket\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.666737 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.666538 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-istio-envoy\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.666737 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.666657 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lht4b\" (UniqueName: \"kubernetes.io/projected/a4215607-32c3-4bf4-8063-a6bf1be43877-kube-api-access-lht4b\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.666737 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.666703 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4215607-32c3-4bf4-8063-a6bf1be43877-istio-token\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.667126 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.666879 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-istio-data\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.667126 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.667067 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-workload-certs\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.667239 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.667215 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-credential-socket\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.667313 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.667293 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-workload-socket\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.667365 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.667330 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4215607-32c3-4bf4-8063-a6bf1be43877-istiod-ca-cert\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.669097 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.669066 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-istio-envoy\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.669355 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.669332 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4215607-32c3-4bf4-8063-a6bf1be43877-istio-podinfo\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.675531 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.675500 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4215607-32c3-4bf4-8063-a6bf1be43877-istio-token\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.675719 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.675587 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lht4b\" (UniqueName: \"kubernetes.io/projected/a4215607-32c3-4bf4-8063-a6bf1be43877-kube-api-access-lht4b\") pod \"e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:31.699867 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.699832 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:48:31.840352 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:31.840313 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:32.031066 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:32.031034 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h"] Jul 21 00:48:32.034465 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:48:32.034434 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4215607_32c3_4bf4_8063_a6bf1be43877.slice/crio-ad347f52291285d45d01a233f06e138dfbea82913e6dce304f1aa9c52f078988 WatchSource:0}: Error finding container ad347f52291285d45d01a233f06e138dfbea82913e6dce304f1aa9c52f078988: Status 404 returned error can't find the container with id ad347f52291285d45d01a233f06e138dfbea82913e6dce304f1aa9c52f078988 Jul 21 00:48:32.036633 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:32.036584 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:48:32.036727 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:32.036681 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:48:32.036727 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:32.036721 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:48:32.148680 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:32.148639 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" event={"ID":"a4215607-32c3-4bf4-8063-a6bf1be43877","Type":"ContainerStarted","Data":"95abb52cb7261a15c6c66b114e6921ba25783d710eca957d742a5751b40ce13f"} Jul 21 00:48:32.148898 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:32.148709 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" event={"ID":"a4215607-32c3-4bf4-8063-a6bf1be43877","Type":"ContainerStarted","Data":"ad347f52291285d45d01a233f06e138dfbea82913e6dce304f1aa9c52f078988"} Jul 21 00:48:32.172158 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:32.172098 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" podStartSLOduration=1.172079559 podStartE2EDuration="1.172079559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:48:32.171485081 +0000 UTC m=+1906.386865406" watchObservedRunningTime="2026-07-21 00:48:32.172079559 +0000 UTC m=+1906.387459885" Jul 21 00:48:32.841108 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:32.841073 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:33.550545 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:33.550509 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:48:33.658514 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:33.658475 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:48:33.750909 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:33.750879 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:48:33.846372 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:33.846345 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:33.863029 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:33.862997 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:48:34.160075 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:34.159989 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:34.161022 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:34.160983 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:34.281885 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:34.281848 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:48:34.388484 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:34.388453 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:48:35.941696 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:35.941669 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:48:36.014128 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.011310 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/174d0146-7c29-44ab-ae3d-579b701131d8-istiod-ca-cert\") pod \"174d0146-7c29-44ab-ae3d-579b701131d8\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " Jul 21 00:48:36.014128 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.011361 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/174d0146-7c29-44ab-ae3d-579b701131d8-istio-podinfo\") pod \"174d0146-7c29-44ab-ae3d-579b701131d8\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " Jul 21 00:48:36.014128 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.011406 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/174d0146-7c29-44ab-ae3d-579b701131d8-istio-token\") pod \"174d0146-7c29-44ab-ae3d-579b701131d8\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " Jul 21 00:48:36.014128 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.011450 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2v8vv\" (UniqueName: \"kubernetes.io/projected/174d0146-7c29-44ab-ae3d-579b701131d8-kube-api-access-2v8vv\") pod \"174d0146-7c29-44ab-ae3d-579b701131d8\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " Jul 21 00:48:36.014128 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.011486 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-istio-data\") pod \"174d0146-7c29-44ab-ae3d-579b701131d8\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " Jul 21 00:48:36.014128 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.011530 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-workload-certs\") pod \"174d0146-7c29-44ab-ae3d-579b701131d8\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " Jul 21 00:48:36.014128 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.011560 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-workload-socket\") pod \"174d0146-7c29-44ab-ae3d-579b701131d8\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " Jul 21 00:48:36.014128 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.011661 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-istio-envoy\") pod \"174d0146-7c29-44ab-ae3d-579b701131d8\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " Jul 21 00:48:36.014128 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.011758 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-credential-socket\") pod \"174d0146-7c29-44ab-ae3d-579b701131d8\" (UID: \"174d0146-7c29-44ab-ae3d-579b701131d8\") " Jul 21 00:48:36.014128 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.012472 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "174d0146-7c29-44ab-ae3d-579b701131d8" (UID: "174d0146-7c29-44ab-ae3d-579b701131d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:48:36.014128 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.012834 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/174d0146-7c29-44ab-ae3d-579b701131d8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "174d0146-7c29-44ab-ae3d-579b701131d8" (UID: "174d0146-7c29-44ab-ae3d-579b701131d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:48:36.015583 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.015549 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/174d0146-7c29-44ab-ae3d-579b701131d8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "174d0146-7c29-44ab-ae3d-579b701131d8" (UID: "174d0146-7c29-44ab-ae3d-579b701131d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 00:48:36.018028 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.017989 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174d0146-7c29-44ab-ae3d-579b701131d8-istio-token" (OuterVolumeSpecName: "istio-token") pod "174d0146-7c29-44ab-ae3d-579b701131d8" (UID: "174d0146-7c29-44ab-ae3d-579b701131d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:48:36.018765 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.018735 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "174d0146-7c29-44ab-ae3d-579b701131d8" (UID: "174d0146-7c29-44ab-ae3d-579b701131d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:48:36.018882 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.018788 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "174d0146-7c29-44ab-ae3d-579b701131d8" (UID: "174d0146-7c29-44ab-ae3d-579b701131d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:48:36.020094 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.020071 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-istio-data" (OuterVolumeSpecName: "istio-data") pod "174d0146-7c29-44ab-ae3d-579b701131d8" (UID: "174d0146-7c29-44ab-ae3d-579b701131d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:48:36.020422 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.020398 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174d0146-7c29-44ab-ae3d-579b701131d8-kube-api-access-2v8vv" (OuterVolumeSpecName: "kube-api-access-2v8vv") pod "174d0146-7c29-44ab-ae3d-579b701131d8" (UID: "174d0146-7c29-44ab-ae3d-579b701131d8"). InnerVolumeSpecName "kube-api-access-2v8vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:48:36.020768 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.020750 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "174d0146-7c29-44ab-ae3d-579b701131d8" (UID: "174d0146-7c29-44ab-ae3d-579b701131d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:48:36.113264 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.113222 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v8vv\" (UniqueName: \"kubernetes.io/projected/174d0146-7c29-44ab-ae3d-579b701131d8-kube-api-access-2v8vv\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:48:36.113264 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.113258 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-istio-data\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:48:36.113264 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.113269 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-workload-certs\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:48:36.113264 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.113277 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-workload-socket\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:48:36.113540 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.113285 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-istio-envoy\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:48:36.113540 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.113293 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/174d0146-7c29-44ab-ae3d-579b701131d8-credential-socket\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:48:36.113540 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.113301 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/174d0146-7c29-44ab-ae3d-579b701131d8-istiod-ca-cert\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:48:36.113540 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.113308 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/174d0146-7c29-44ab-ae3d-579b701131d8-istio-podinfo\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:48:36.113540 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.113315 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/174d0146-7c29-44ab-ae3d-579b701131d8-istio-token\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:48:36.173519 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.173481 2580 generic.go:356] "Generic (PLEG): container finished" podID="174d0146-7c29-44ab-ae3d-579b701131d8" containerID="892441250fe59d6b73e4f4fc94d519e3551d1fce4a9e4039950039d47c84ecc0" exitCode=0 Jul 21 00:48:36.173727 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.173558 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" Jul 21 00:48:36.173727 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.173575 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" event={"ID":"174d0146-7c29-44ab-ae3d-579b701131d8","Type":"ContainerDied","Data":"892441250fe59d6b73e4f4fc94d519e3551d1fce4a9e4039950039d47c84ecc0"} Jul 21 00:48:36.173727 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.173690 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch" event={"ID":"174d0146-7c29-44ab-ae3d-579b701131d8","Type":"ContainerDied","Data":"fbcf56aa72bccd279cfb0a0ff3eca2cd56d8a933639ad0426304f270734ead54"} Jul 21 00:48:36.173727 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.173724 2580 scope.go:117] "RemoveContainer" containerID="892441250fe59d6b73e4f4fc94d519e3551d1fce4a9e4039950039d47c84ecc0" Jul 21 00:48:36.186398 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.186370 2580 scope.go:117] "RemoveContainer" containerID="dde8179fb7abde097c13372cd329279e5550cc4342ef15afcfaefe5a92d28ae7" Jul 21 00:48:36.196357 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.196331 2580 scope.go:117] "RemoveContainer" containerID="892441250fe59d6b73e4f4fc94d519e3551d1fce4a9e4039950039d47c84ecc0" Jul 21 00:48:36.196726 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:48:36.196703 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"892441250fe59d6b73e4f4fc94d519e3551d1fce4a9e4039950039d47c84ecc0\": container with ID starting with 892441250fe59d6b73e4f4fc94d519e3551d1fce4a9e4039950039d47c84ecc0 not found: ID does not exist" containerID="892441250fe59d6b73e4f4fc94d519e3551d1fce4a9e4039950039d47c84ecc0" Jul 21 00:48:36.196803 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.196736 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"892441250fe59d6b73e4f4fc94d519e3551d1fce4a9e4039950039d47c84ecc0"} err="failed to get container status \"892441250fe59d6b73e4f4fc94d519e3551d1fce4a9e4039950039d47c84ecc0\": rpc error: code = NotFound desc = could not find container \"892441250fe59d6b73e4f4fc94d519e3551d1fce4a9e4039950039d47c84ecc0\": container with ID starting with 892441250fe59d6b73e4f4fc94d519e3551d1fce4a9e4039950039d47c84ecc0 not found: ID does not exist" Jul 21 00:48:36.196803 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.196768 2580 scope.go:117] "RemoveContainer" containerID="dde8179fb7abde097c13372cd329279e5550cc4342ef15afcfaefe5a92d28ae7" Jul 21 00:48:36.197037 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:48:36.197019 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dde8179fb7abde097c13372cd329279e5550cc4342ef15afcfaefe5a92d28ae7\": container with ID starting with dde8179fb7abde097c13372cd329279e5550cc4342ef15afcfaefe5a92d28ae7 not found: ID does not exist" containerID="dde8179fb7abde097c13372cd329279e5550cc4342ef15afcfaefe5a92d28ae7" Jul 21 00:48:36.197091 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.197047 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dde8179fb7abde097c13372cd329279e5550cc4342ef15afcfaefe5a92d28ae7"} err="failed to get container status \"dde8179fb7abde097c13372cd329279e5550cc4342ef15afcfaefe5a92d28ae7\": rpc error: code = NotFound desc = could not find container \"dde8179fb7abde097c13372cd329279e5550cc4342ef15afcfaefe5a92d28ae7\": container with ID starting with dde8179fb7abde097c13372cd329279e5550cc4342ef15afcfaefe5a92d28ae7 not found: ID does not exist" Jul 21 00:48:36.202834 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.202805 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch"] Jul 21 00:48:36.207565 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.207533 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-c7cbed04-gw-openshift-default-56f955c85f-hxtch"] Jul 21 00:48:36.299197 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:36.299119 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="174d0146-7c29-44ab-ae3d-579b701131d8" path="/var/lib/kubelet/pods/174d0146-7c29-44ab-ae3d-579b701131d8/volumes" Jul 21 00:48:48.253259 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:48.253225 2580 generic.go:356] "Generic (PLEG): container finished" podID="a4215607-32c3-4bf4-8063-a6bf1be43877" containerID="95abb52cb7261a15c6c66b114e6921ba25783d710eca957d742a5751b40ce13f" exitCode=0 Jul 21 00:48:48.253781 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:48.253304 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" event={"ID":"a4215607-32c3-4bf4-8063-a6bf1be43877","Type":"ContainerDied","Data":"95abb52cb7261a15c6c66b114e6921ba25783d710eca957d742a5751b40ce13f"} Jul 21 00:48:48.253781 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:48.253713 2580 scope.go:117] "RemoveContainer" containerID="95abb52cb7261a15c6c66b114e6921ba25783d710eca957d742a5751b40ce13f" Jul 21 00:48:48.254794 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:48.254762 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:48:48.254893 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:48.254874 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:48:48.254946 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:48.254922 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:48:49.160661 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:49.160590 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:49.261048 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:49.261011 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" event={"ID":"a4215607-32c3-4bf4-8063-a6bf1be43877","Type":"ContainerStarted","Data":"014cdfe4f7fdb525b2d422dd7f4587023041efd7a45b3f0f2ca75e75c50a1a26"} Jul 21 00:48:49.841435 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:49.841394 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:49.842565 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:49.842541 2580 patch_prober.go:28] interesting pod/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 21 00:48:49.842636 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:49.842610 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" podUID="a4215607-32c3-4bf4-8063-a6bf1be43877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 21 00:48:50.841098 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:50.841057 2580 patch_prober.go:28] interesting pod/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 21 00:48:50.841493 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:50.841132 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" podUID="a4215607-32c3-4bf4-8063-a6bf1be43877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 21 00:48:51.841469 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:51.841429 2580 patch_prober.go:28] interesting pod/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 21 00:48:51.841911 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:51.841492 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" podUID="a4215607-32c3-4bf4-8063-a6bf1be43877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 21 00:48:52.841729 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:52.841685 2580 patch_prober.go:28] interesting pod/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 21 00:48:52.842209 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:52.841758 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" podUID="a4215607-32c3-4bf4-8063-a6bf1be43877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 21 00:48:53.845176 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:53.845138 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:53.845704 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:53.845616 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:48:53.846331 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:48:53.846313 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:49:10.021143 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:10.021109 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:49:14.515076 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:14.515039 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:49:15.763436 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:15.763400 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h"] Jul 21 00:49:15.764060 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:15.763650 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" podUID="a4215607-32c3-4bf4-8063-a6bf1be43877" containerName="istio-proxy" containerID="cri-o://014cdfe4f7fdb525b2d422dd7f4587023041efd7a45b3f0f2ca75e75c50a1a26" gracePeriod=30 Jul 21 00:49:15.786436 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:15.786402 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:49:16.035934 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.035828 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:49:16.278491 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.278455 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:49:16.578389 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.578355 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8"] Jul 21 00:49:16.578906 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.578886 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="174d0146-7c29-44ab-ae3d-579b701131d8" containerName="istio-proxy" Jul 21 00:49:16.578906 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.578907 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="174d0146-7c29-44ab-ae3d-579b701131d8" containerName="istio-proxy" Jul 21 00:49:16.579073 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.578933 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="174d0146-7c29-44ab-ae3d-579b701131d8" containerName="istio-proxy" Jul 21 00:49:16.579073 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.578941 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="174d0146-7c29-44ab-ae3d-579b701131d8" containerName="istio-proxy" Jul 21 00:49:16.579073 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.579045 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="174d0146-7c29-44ab-ae3d-579b701131d8" containerName="istio-proxy" Jul 21 00:49:16.579073 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.579059 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="174d0146-7c29-44ab-ae3d-579b701131d8" containerName="istio-proxy" Jul 21 00:49:16.582130 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.582105 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.584777 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.584753 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-e0357210-openshift-default-dockercfg-9jkf8\"" Jul 21 00:49:16.593663 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.593633 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8"] Jul 21 00:49:16.604015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.603982 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:49:16.607317 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.607289 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-credential-socket\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.607487 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.607349 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-istio-envoy\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.607487 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.607376 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d9186ec-24ac-46c3-a214-d8b9c8025672-istio-token\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.607487 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.607442 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-istio-data\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.607487 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.607467 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-workload-socket\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.607723 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.607503 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-workload-certs\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.607723 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.607527 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d9186ec-24ac-46c3-a214-d8b9c8025672-istiod-ca-cert\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.607723 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.607623 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzmws\" (UniqueName: \"kubernetes.io/projected/6d9186ec-24ac-46c3-a214-d8b9c8025672-kube-api-access-pzmws\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.607723 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.607652 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d9186ec-24ac-46c3-a214-d8b9c8025672-istio-podinfo\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.708977 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.708933 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-credential-socket\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.709190 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.708991 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-istio-envoy\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.709190 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.709108 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d9186ec-24ac-46c3-a214-d8b9c8025672-istio-token\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.709309 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.709202 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-istio-data\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.709309 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.709235 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-workload-socket\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.709309 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.709277 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-workload-certs\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.709528 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.709506 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-credential-socket\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.709620 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.709561 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d9186ec-24ac-46c3-a214-d8b9c8025672-istiod-ca-cert\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.709683 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.709633 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-workload-socket\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.709683 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.709654 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-istio-data\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.709785 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.709688 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzmws\" (UniqueName: \"kubernetes.io/projected/6d9186ec-24ac-46c3-a214-d8b9c8025672-kube-api-access-pzmws\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.709785 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.709736 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-workload-certs\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.709893 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.709878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d9186ec-24ac-46c3-a214-d8b9c8025672-istio-podinfo\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.710018 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.709997 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d9186ec-24ac-46c3-a214-d8b9c8025672-istiod-ca-cert\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.711457 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.711424 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-istio-envoy\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.712267 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.712245 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d9186ec-24ac-46c3-a214-d8b9c8025672-istio-podinfo\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.718147 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.718118 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzmws\" (UniqueName: \"kubernetes.io/projected/6d9186ec-24ac-46c3-a214-d8b9c8025672-kube-api-access-pzmws\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.718871 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.718848 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d9186ec-24ac-46c3-a214-d8b9c8025672-istio-token\") pod \"e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:16.771130 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.771097 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:49:16.894201 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:16.894106 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:17.042671 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:17.042640 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8"] Jul 21 00:49:17.044372 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:49:17.044344 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d9186ec_24ac_46c3_a214_d8b9c8025672.slice/crio-81d740361f7eeebe0f852841c3f3e6c34173e90f484e4842fa25bdd8fe3caa9d WatchSource:0}: Error finding container 81d740361f7eeebe0f852841c3f3e6c34173e90f484e4842fa25bdd8fe3caa9d: Status 404 returned error can't find the container with id 81d740361f7eeebe0f852841c3f3e6c34173e90f484e4842fa25bdd8fe3caa9d Jul 21 00:49:17.047071 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:17.047040 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:49:17.047147 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:17.047110 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:49:17.047147 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:17.047136 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:49:17.451404 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:17.450729 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" event={"ID":"6d9186ec-24ac-46c3-a214-d8b9c8025672","Type":"ContainerStarted","Data":"b68e9cdbda15998c629954535eed14a15abb26970c83eabfe59cbd35729744d4"} Jul 21 00:49:17.451404 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:17.450798 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" event={"ID":"6d9186ec-24ac-46c3-a214-d8b9c8025672","Type":"ContainerStarted","Data":"81d740361f7eeebe0f852841c3f3e6c34173e90f484e4842fa25bdd8fe3caa9d"} Jul 21 00:49:17.473441 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:17.473386 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" podStartSLOduration=1.473368826 podStartE2EDuration="1.473368826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:49:17.470676607 +0000 UTC m=+1951.686056956" watchObservedRunningTime="2026-07-21 00:49:17.473368826 +0000 UTC m=+1951.688749150" Jul 21 00:49:17.894796 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:17.894759 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:18.809857 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:18.809824 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:49:18.900260 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:18.900229 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:18.901726 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:18.901700 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:49:19.033070 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:19.033036 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:49:19.187816 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:19.187706 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:49:19.462419 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:19.462329 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:19.463377 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:19.463358 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:19.556471 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:19.556427 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:49:19.650895 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:19.650824 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:49:21.028236 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.028167 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:49:21.050327 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.050291 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4215607-32c3-4bf4-8063-a6bf1be43877-istiod-ca-cert\") pod \"a4215607-32c3-4bf4-8063-a6bf1be43877\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " Jul 21 00:49:21.050535 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.050344 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-workload-certs\") pod \"a4215607-32c3-4bf4-8063-a6bf1be43877\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " Jul 21 00:49:21.050535 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.050381 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4215607-32c3-4bf4-8063-a6bf1be43877-istio-podinfo\") pod \"a4215607-32c3-4bf4-8063-a6bf1be43877\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " Jul 21 00:49:21.050535 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.050396 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-credential-socket\") pod \"a4215607-32c3-4bf4-8063-a6bf1be43877\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " Jul 21 00:49:21.050535 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.050515 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-istio-data\") pod \"a4215607-32c3-4bf4-8063-a6bf1be43877\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " Jul 21 00:49:21.050798 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.050564 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-istio-envoy\") pod \"a4215607-32c3-4bf4-8063-a6bf1be43877\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " Jul 21 00:49:21.050798 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.050643 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4215607-32c3-4bf4-8063-a6bf1be43877-istio-token\") pod \"a4215607-32c3-4bf4-8063-a6bf1be43877\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " Jul 21 00:49:21.050798 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.050684 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lht4b\" (UniqueName: \"kubernetes.io/projected/a4215607-32c3-4bf4-8063-a6bf1be43877-kube-api-access-lht4b\") pod \"a4215607-32c3-4bf4-8063-a6bf1be43877\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " Jul 21 00:49:21.050798 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.050711 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-workload-socket\") pod \"a4215607-32c3-4bf4-8063-a6bf1be43877\" (UID: \"a4215607-32c3-4bf4-8063-a6bf1be43877\") " Jul 21 00:49:21.050798 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.050727 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a4215607-32c3-4bf4-8063-a6bf1be43877" (UID: "a4215607-32c3-4bf4-8063-a6bf1be43877"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:49:21.051043 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.050854 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4215607-32c3-4bf4-8063-a6bf1be43877-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a4215607-32c3-4bf4-8063-a6bf1be43877" (UID: "a4215607-32c3-4bf4-8063-a6bf1be43877"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:49:21.051097 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.051023 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a4215607-32c3-4bf4-8063-a6bf1be43877" (UID: "a4215607-32c3-4bf4-8063-a6bf1be43877"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:49:21.051097 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.051055 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a4215607-32c3-4bf4-8063-a6bf1be43877" (UID: "a4215607-32c3-4bf4-8063-a6bf1be43877"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:49:21.051198 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.051160 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-workload-certs\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:49:21.051198 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.051180 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-credential-socket\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:49:21.051198 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.051195 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-workload-socket\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:49:21.051339 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.051209 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4215607-32c3-4bf4-8063-a6bf1be43877-istiod-ca-cert\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:49:21.051708 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.051682 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-istio-data" (OuterVolumeSpecName: "istio-data") pod "a4215607-32c3-4bf4-8063-a6bf1be43877" (UID: "a4215607-32c3-4bf4-8063-a6bf1be43877"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:49:21.053257 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.053223 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4215607-32c3-4bf4-8063-a6bf1be43877-istio-token" (OuterVolumeSpecName: "istio-token") pod "a4215607-32c3-4bf4-8063-a6bf1be43877" (UID: "a4215607-32c3-4bf4-8063-a6bf1be43877"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:49:21.053667 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.053634 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4215607-32c3-4bf4-8063-a6bf1be43877-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a4215607-32c3-4bf4-8063-a6bf1be43877" (UID: "a4215607-32c3-4bf4-8063-a6bf1be43877"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 00:49:21.053667 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.053647 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4215607-32c3-4bf4-8063-a6bf1be43877-kube-api-access-lht4b" (OuterVolumeSpecName: "kube-api-access-lht4b") pod "a4215607-32c3-4bf4-8063-a6bf1be43877" (UID: "a4215607-32c3-4bf4-8063-a6bf1be43877"). InnerVolumeSpecName "kube-api-access-lht4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:49:21.053805 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.053711 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a4215607-32c3-4bf4-8063-a6bf1be43877" (UID: "a4215607-32c3-4bf4-8063-a6bf1be43877"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:49:21.152026 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.151984 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4215607-32c3-4bf4-8063-a6bf1be43877-istio-podinfo\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:49:21.152026 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.152020 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-istio-data\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:49:21.152026 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.152029 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4215607-32c3-4bf4-8063-a6bf1be43877-istio-envoy\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:49:21.152026 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.152037 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4215607-32c3-4bf4-8063-a6bf1be43877-istio-token\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:49:21.152322 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.152046 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lht4b\" (UniqueName: \"kubernetes.io/projected/a4215607-32c3-4bf4-8063-a6bf1be43877-kube-api-access-lht4b\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:49:21.475214 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.475180 2580 generic.go:356] "Generic (PLEG): container finished" podID="a4215607-32c3-4bf4-8063-a6bf1be43877" containerID="014cdfe4f7fdb525b2d422dd7f4587023041efd7a45b3f0f2ca75e75c50a1a26" exitCode=0 Jul 21 00:49:21.475370 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.475226 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" Jul 21 00:49:21.475370 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.475261 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" event={"ID":"a4215607-32c3-4bf4-8063-a6bf1be43877","Type":"ContainerDied","Data":"014cdfe4f7fdb525b2d422dd7f4587023041efd7a45b3f0f2ca75e75c50a1a26"} Jul 21 00:49:21.475370 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.475315 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h" event={"ID":"a4215607-32c3-4bf4-8063-a6bf1be43877","Type":"ContainerDied","Data":"ad347f52291285d45d01a233f06e138dfbea82913e6dce304f1aa9c52f078988"} Jul 21 00:49:21.475370 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.475339 2580 scope.go:117] "RemoveContainer" containerID="014cdfe4f7fdb525b2d422dd7f4587023041efd7a45b3f0f2ca75e75c50a1a26" Jul 21 00:49:21.488076 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.488051 2580 scope.go:117] "RemoveContainer" containerID="95abb52cb7261a15c6c66b114e6921ba25783d710eca957d742a5751b40ce13f" Jul 21 00:49:21.497533 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.497506 2580 scope.go:117] "RemoveContainer" containerID="014cdfe4f7fdb525b2d422dd7f4587023041efd7a45b3f0f2ca75e75c50a1a26" Jul 21 00:49:21.497868 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:49:21.497846 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"014cdfe4f7fdb525b2d422dd7f4587023041efd7a45b3f0f2ca75e75c50a1a26\": container with ID starting with 014cdfe4f7fdb525b2d422dd7f4587023041efd7a45b3f0f2ca75e75c50a1a26 not found: ID does not exist" containerID="014cdfe4f7fdb525b2d422dd7f4587023041efd7a45b3f0f2ca75e75c50a1a26" Jul 21 00:49:21.497919 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.497881 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"014cdfe4f7fdb525b2d422dd7f4587023041efd7a45b3f0f2ca75e75c50a1a26"} err="failed to get container status \"014cdfe4f7fdb525b2d422dd7f4587023041efd7a45b3f0f2ca75e75c50a1a26\": rpc error: code = NotFound desc = could not find container \"014cdfe4f7fdb525b2d422dd7f4587023041efd7a45b3f0f2ca75e75c50a1a26\": container with ID starting with 014cdfe4f7fdb525b2d422dd7f4587023041efd7a45b3f0f2ca75e75c50a1a26 not found: ID does not exist" Jul 21 00:49:21.497919 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.497906 2580 scope.go:117] "RemoveContainer" containerID="95abb52cb7261a15c6c66b114e6921ba25783d710eca957d742a5751b40ce13f" Jul 21 00:49:21.498209 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:49:21.498190 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95abb52cb7261a15c6c66b114e6921ba25783d710eca957d742a5751b40ce13f\": container with ID starting with 95abb52cb7261a15c6c66b114e6921ba25783d710eca957d742a5751b40ce13f not found: ID does not exist" containerID="95abb52cb7261a15c6c66b114e6921ba25783d710eca957d742a5751b40ce13f" Jul 21 00:49:21.498252 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.498217 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95abb52cb7261a15c6c66b114e6921ba25783d710eca957d742a5751b40ce13f"} err="failed to get container status \"95abb52cb7261a15c6c66b114e6921ba25783d710eca957d742a5751b40ce13f\": rpc error: code = NotFound desc = could not find container \"95abb52cb7261a15c6c66b114e6921ba25783d710eca957d742a5751b40ce13f\": container with ID starting with 95abb52cb7261a15c6c66b114e6921ba25783d710eca957d742a5751b40ce13f not found: ID does not exist" Jul 21 00:49:21.498393 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.498377 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h"] Jul 21 00:49:21.502023 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:21.501998 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e23578bb-openshift-default-858b5dc899-spt6h"] Jul 21 00:49:22.297949 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:22.297918 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4215607-32c3-4bf4-8063-a6bf1be43877" path="/var/lib/kubelet/pods/a4215607-32c3-4bf4-8063-a6bf1be43877/volumes" Jul 21 00:49:32.548361 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:32.548254 2580 generic.go:356] "Generic (PLEG): container finished" podID="6d9186ec-24ac-46c3-a214-d8b9c8025672" containerID="b68e9cdbda15998c629954535eed14a15abb26970c83eabfe59cbd35729744d4" exitCode=0 Jul 21 00:49:32.548361 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:32.548322 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" event={"ID":"6d9186ec-24ac-46c3-a214-d8b9c8025672","Type":"ContainerDied","Data":"b68e9cdbda15998c629954535eed14a15abb26970c83eabfe59cbd35729744d4"} Jul 21 00:49:32.548872 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:32.548774 2580 scope.go:117] "RemoveContainer" containerID="b68e9cdbda15998c629954535eed14a15abb26970c83eabfe59cbd35729744d4" Jul 21 00:49:32.549837 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:32.549809 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:49:32.549917 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:32.549894 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:49:32.549966 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:32.549938 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:49:33.555887 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:33.555846 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" event={"ID":"6d9186ec-24ac-46c3-a214-d8b9c8025672","Type":"ContainerStarted","Data":"c0e70a760f5554a3e718b6a6de1911d9bf130c1f215385c9bd5a2d478e180cf8"} Jul 21 00:49:33.625454 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:33.625423 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:49:33.728158 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:33.728126 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:49:33.894760 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:33.894648 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:33.896177 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:33.896138 2580 patch_prober.go:28] interesting pod/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 21 00:49:33.896358 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:33.896229 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" podUID="6d9186ec-24ac-46c3-a214-d8b9c8025672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 21 00:49:34.462473 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:34.462434 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:34.895054 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:34.894990 2580 patch_prober.go:28] interesting pod/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 21 00:49:34.895566 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:34.895108 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" podUID="6d9186ec-24ac-46c3-a214-d8b9c8025672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 21 00:49:35.895231 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:35.894886 2580 patch_prober.go:28] interesting pod/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 21 00:49:35.895231 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:35.894951 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" podUID="6d9186ec-24ac-46c3-a214-d8b9c8025672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 21 00:49:36.912261 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:36.912146 2580 patch_prober.go:28] interesting pod/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 21 00:49:36.912261 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:36.912230 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" podUID="6d9186ec-24ac-46c3-a214-d8b9c8025672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 21 00:49:37.898709 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:37.898673 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:37.899744 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:37.899722 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:49:40.395666 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:40.395591 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:49:40.508536 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:40.508494 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:49:54.305578 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:54.305540 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:49:58.827035 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:58.826996 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:49:58.983204 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:58.983168 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8"] Jul 21 00:49:58.983420 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:58.983398 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" podUID="6d9186ec-24ac-46c3-a214-d8b9c8025672" containerName="istio-proxy" containerID="cri-o://c0e70a760f5554a3e718b6a6de1911d9bf130c1f215385c9bd5a2d478e180cf8" gracePeriod=30 Jul 21 00:49:59.002460 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:59.002432 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:49:59.269988 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:59.269890 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:49:59.777146 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:59.777104 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:49:59.877922 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:59.877871 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:49:59.972887 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:49:59.972851 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:50:01.442876 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:01.442844 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:50:04.240329 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.240298 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:50:04.277640 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.277541 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-istio-data\") pod \"6d9186ec-24ac-46c3-a214-d8b9c8025672\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " Jul 21 00:50:04.277640 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.277576 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d9186ec-24ac-46c3-a214-d8b9c8025672-istiod-ca-cert\") pod \"6d9186ec-24ac-46c3-a214-d8b9c8025672\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " Jul 21 00:50:04.277640 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.277595 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzmws\" (UniqueName: \"kubernetes.io/projected/6d9186ec-24ac-46c3-a214-d8b9c8025672-kube-api-access-pzmws\") pod \"6d9186ec-24ac-46c3-a214-d8b9c8025672\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " Jul 21 00:50:04.277640 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.277628 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-credential-socket\") pod \"6d9186ec-24ac-46c3-a214-d8b9c8025672\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " Jul 21 00:50:04.277640 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.277644 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d9186ec-24ac-46c3-a214-d8b9c8025672-istio-podinfo\") pod \"6d9186ec-24ac-46c3-a214-d8b9c8025672\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " Jul 21 00:50:04.278030 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.277702 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d9186ec-24ac-46c3-a214-d8b9c8025672-istio-token\") pod \"6d9186ec-24ac-46c3-a214-d8b9c8025672\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " Jul 21 00:50:04.278030 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.277720 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-istio-envoy\") pod \"6d9186ec-24ac-46c3-a214-d8b9c8025672\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " Jul 21 00:50:04.278030 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.277740 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-workload-certs\") pod \"6d9186ec-24ac-46c3-a214-d8b9c8025672\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " Jul 21 00:50:04.278030 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.277765 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-workload-socket\") pod \"6d9186ec-24ac-46c3-a214-d8b9c8025672\" (UID: \"6d9186ec-24ac-46c3-a214-d8b9c8025672\") " Jul 21 00:50:04.278030 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.277882 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6d9186ec-24ac-46c3-a214-d8b9c8025672" (UID: "6d9186ec-24ac-46c3-a214-d8b9c8025672"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:50:04.278276 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.278058 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-credential-socket\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:50:04.278276 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.278128 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6d9186ec-24ac-46c3-a214-d8b9c8025672" (UID: "6d9186ec-24ac-46c3-a214-d8b9c8025672"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:50:04.278276 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.278036 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d9186ec-24ac-46c3-a214-d8b9c8025672-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6d9186ec-24ac-46c3-a214-d8b9c8025672" (UID: "6d9186ec-24ac-46c3-a214-d8b9c8025672"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:50:04.278276 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.278253 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6d9186ec-24ac-46c3-a214-d8b9c8025672" (UID: "6d9186ec-24ac-46c3-a214-d8b9c8025672"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:50:04.278825 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.278774 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-istio-data" (OuterVolumeSpecName: "istio-data") pod "6d9186ec-24ac-46c3-a214-d8b9c8025672" (UID: "6d9186ec-24ac-46c3-a214-d8b9c8025672"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:50:04.280216 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.280183 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d9186ec-24ac-46c3-a214-d8b9c8025672-kube-api-access-pzmws" (OuterVolumeSpecName: "kube-api-access-pzmws") pod "6d9186ec-24ac-46c3-a214-d8b9c8025672" (UID: "6d9186ec-24ac-46c3-a214-d8b9c8025672"). InnerVolumeSpecName "kube-api-access-pzmws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:50:04.280335 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.280210 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6d9186ec-24ac-46c3-a214-d8b9c8025672" (UID: "6d9186ec-24ac-46c3-a214-d8b9c8025672"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:50:04.280335 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.280296 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d9186ec-24ac-46c3-a214-d8b9c8025672-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6d9186ec-24ac-46c3-a214-d8b9c8025672" (UID: "6d9186ec-24ac-46c3-a214-d8b9c8025672"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 00:50:04.280654 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.280631 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d9186ec-24ac-46c3-a214-d8b9c8025672-istio-token" (OuterVolumeSpecName: "istio-token") pod "6d9186ec-24ac-46c3-a214-d8b9c8025672" (UID: "6d9186ec-24ac-46c3-a214-d8b9c8025672"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:50:04.379636 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.379566 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-istio-data\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:50:04.379636 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.379630 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d9186ec-24ac-46c3-a214-d8b9c8025672-istiod-ca-cert\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:50:04.379636 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.379644 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzmws\" (UniqueName: \"kubernetes.io/projected/6d9186ec-24ac-46c3-a214-d8b9c8025672-kube-api-access-pzmws\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:50:04.379890 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.379655 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d9186ec-24ac-46c3-a214-d8b9c8025672-istio-podinfo\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:50:04.379890 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.379667 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d9186ec-24ac-46c3-a214-d8b9c8025672-istio-token\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:50:04.379890 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.379677 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-istio-envoy\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:50:04.379890 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.379689 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-workload-certs\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:50:04.379890 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.379700 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9186ec-24ac-46c3-a214-d8b9c8025672-workload-socket\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:50:04.736892 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.736858 2580 generic.go:356] "Generic (PLEG): container finished" podID="6d9186ec-24ac-46c3-a214-d8b9c8025672" containerID="c0e70a760f5554a3e718b6a6de1911d9bf130c1f215385c9bd5a2d478e180cf8" exitCode=0 Jul 21 00:50:04.737051 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.736914 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" Jul 21 00:50:04.737051 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.736917 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" event={"ID":"6d9186ec-24ac-46c3-a214-d8b9c8025672","Type":"ContainerDied","Data":"c0e70a760f5554a3e718b6a6de1911d9bf130c1f215385c9bd5a2d478e180cf8"} Jul 21 00:50:04.737051 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.737034 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8" event={"ID":"6d9186ec-24ac-46c3-a214-d8b9c8025672","Type":"ContainerDied","Data":"81d740361f7eeebe0f852841c3f3e6c34173e90f484e4842fa25bdd8fe3caa9d"} Jul 21 00:50:04.737206 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.737056 2580 scope.go:117] "RemoveContainer" containerID="c0e70a760f5554a3e718b6a6de1911d9bf130c1f215385c9bd5a2d478e180cf8" Jul 21 00:50:04.746349 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.746265 2580 scope.go:117] "RemoveContainer" containerID="b68e9cdbda15998c629954535eed14a15abb26970c83eabfe59cbd35729744d4" Jul 21 00:50:04.755227 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.755202 2580 scope.go:117] "RemoveContainer" containerID="c0e70a760f5554a3e718b6a6de1911d9bf130c1f215385c9bd5a2d478e180cf8" Jul 21 00:50:04.755585 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:50:04.755559 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0e70a760f5554a3e718b6a6de1911d9bf130c1f215385c9bd5a2d478e180cf8\": container with ID starting with c0e70a760f5554a3e718b6a6de1911d9bf130c1f215385c9bd5a2d478e180cf8 not found: ID does not exist" containerID="c0e70a760f5554a3e718b6a6de1911d9bf130c1f215385c9bd5a2d478e180cf8" Jul 21 00:50:04.755710 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.755592 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0e70a760f5554a3e718b6a6de1911d9bf130c1f215385c9bd5a2d478e180cf8"} err="failed to get container status \"c0e70a760f5554a3e718b6a6de1911d9bf130c1f215385c9bd5a2d478e180cf8\": rpc error: code = NotFound desc = could not find container \"c0e70a760f5554a3e718b6a6de1911d9bf130c1f215385c9bd5a2d478e180cf8\": container with ID starting with c0e70a760f5554a3e718b6a6de1911d9bf130c1f215385c9bd5a2d478e180cf8 not found: ID does not exist" Jul 21 00:50:04.755710 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.755635 2580 scope.go:117] "RemoveContainer" containerID="b68e9cdbda15998c629954535eed14a15abb26970c83eabfe59cbd35729744d4" Jul 21 00:50:04.755966 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:50:04.755922 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68e9cdbda15998c629954535eed14a15abb26970c83eabfe59cbd35729744d4\": container with ID starting with b68e9cdbda15998c629954535eed14a15abb26970c83eabfe59cbd35729744d4 not found: ID does not exist" containerID="b68e9cdbda15998c629954535eed14a15abb26970c83eabfe59cbd35729744d4" Jul 21 00:50:04.755966 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.755948 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68e9cdbda15998c629954535eed14a15abb26970c83eabfe59cbd35729744d4"} err="failed to get container status \"b68e9cdbda15998c629954535eed14a15abb26970c83eabfe59cbd35729744d4\": rpc error: code = NotFound desc = could not find container \"b68e9cdbda15998c629954535eed14a15abb26970c83eabfe59cbd35729744d4\": container with ID starting with b68e9cdbda15998c629954535eed14a15abb26970c83eabfe59cbd35729744d4 not found: ID does not exist" Jul 21 00:50:04.758145 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.758119 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8"] Jul 21 00:50:04.761560 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:04.761533 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-e0357210-openshift-default-55679d9fbc-t6bf8"] Jul 21 00:50:06.297957 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:06.297919 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d9186ec-24ac-46c3-a214-d8b9c8025672" path="/var/lib/kubelet/pods/6d9186ec-24ac-46c3-a214-d8b9c8025672/volumes" Jul 21 00:50:07.339468 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:07.339427 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:50:07.443197 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:07.443155 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:50:07.575074 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:07.575041 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:50:07.716296 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:07.716197 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:50:08.122258 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:08.122223 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:50:14.946900 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:14.946865 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:50:15.001088 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:15.001051 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:50:26.669288 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:26.669248 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:50:26.768846 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:26.768809 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:50:34.136453 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:34.136415 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:50:40.626060 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:40.626027 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:50:40.725085 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:40.725042 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:50:40.825162 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:40.825128 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:50:56.873247 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:50:56.873213 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:02.582517 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:02.582479 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:02.799688 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:02.799652 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:02.921819 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:02.921724 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:13.310356 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:13.310269 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:14.161831 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:14.161789 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:20.210640 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:20.210574 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:20.316720 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:20.316684 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:20.413294 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:20.413266 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:20.514504 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:20.514417 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:21.379461 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:21.379421 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:21.481975 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:21.481943 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:21.581946 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:21.581910 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:23.344801 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:23.344771 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:29.178092 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:29.178051 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:29.278130 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:29.278097 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:29.893871 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:29.893824 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:35.693002 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:35.692965 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:35.790386 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:35.790351 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:35.943070 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:35.942990 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:36.256290 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.256199 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd"] Jul 21 00:51:36.256710 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.256694 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4215607-32c3-4bf4-8063-a6bf1be43877" containerName="istio-proxy" Jul 21 00:51:36.256767 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.256711 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4215607-32c3-4bf4-8063-a6bf1be43877" containerName="istio-proxy" Jul 21 00:51:36.256767 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.256725 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d9186ec-24ac-46c3-a214-d8b9c8025672" containerName="istio-proxy" Jul 21 00:51:36.256767 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.256730 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d9186ec-24ac-46c3-a214-d8b9c8025672" containerName="istio-proxy" Jul 21 00:51:36.256767 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.256742 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d9186ec-24ac-46c3-a214-d8b9c8025672" containerName="istio-proxy" Jul 21 00:51:36.256767 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.256750 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d9186ec-24ac-46c3-a214-d8b9c8025672" containerName="istio-proxy" Jul 21 00:51:36.256767 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.256761 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4215607-32c3-4bf4-8063-a6bf1be43877" containerName="istio-proxy" Jul 21 00:51:36.256767 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.256766 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4215607-32c3-4bf4-8063-a6bf1be43877" containerName="istio-proxy" Jul 21 00:51:36.256970 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.256848 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d9186ec-24ac-46c3-a214-d8b9c8025672" containerName="istio-proxy" Jul 21 00:51:36.256970 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.256858 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4215607-32c3-4bf4-8063-a6bf1be43877" containerName="istio-proxy" Jul 21 00:51:36.256970 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.256866 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4215607-32c3-4bf4-8063-a6bf1be43877" containerName="istio-proxy" Jul 21 00:51:36.260112 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.260092 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.262673 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.262651 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8c05e2d2-openshift-default-dockercfg-lqv2v\"" Jul 21 00:51:36.282565 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.282533 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd"] Jul 21 00:51:36.285529 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.285499 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:36.363137 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.363094 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-credential-socket\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.363137 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.363144 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-workload-socket\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.363400 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.363162 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istiod-ca-cert\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.363400 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.363245 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj9zw\" (UniqueName: \"kubernetes.io/projected/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-kube-api-access-qj9zw\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.363400 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.363331 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istio-envoy\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.363400 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.363373 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istio-data\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.363400 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.363397 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istio-podinfo\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.363700 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.363521 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istio-token\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.363700 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.363557 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-workload-certs\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.377133 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.377105 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:36.464572 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.464531 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-credential-socket\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.464572 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.464576 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-workload-socket\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.464875 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.464592 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istiod-ca-cert\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.464875 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.464631 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj9zw\" (UniqueName: \"kubernetes.io/projected/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-kube-api-access-qj9zw\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.464875 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.464671 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istio-envoy\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.464875 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.464706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istio-data\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.464875 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.464729 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istio-podinfo\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.464875 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.464753 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istio-token\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.464875 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.464785 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-workload-certs\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.465231 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.465075 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-workload-socket\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.465231 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.465140 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-credential-socket\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.465231 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.465167 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istio-data\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.465231 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.465202 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-workload-certs\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.465467 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.465443 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istiod-ca-cert\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.467372 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.467343 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istio-podinfo\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.467372 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.467352 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istio-envoy\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.472507 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.472482 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istio-token\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.472831 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.472811 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj9zw\" (UniqueName: \"kubernetes.io/projected/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-kube-api-access-qj9zw\") pod \"e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.570140 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.570094 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:36.717824 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.717794 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd"] Jul 21 00:51:36.719633 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:51:36.719585 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d98c01e_14c5_402a_b38d_f1ddd9a408e4.slice/crio-0f1ca4f224834df51ba490d0f5acd1f8e34ffb0f0c4a03a138ed45db3f8eeefe WatchSource:0}: Error finding container 0f1ca4f224834df51ba490d0f5acd1f8e34ffb0f0c4a03a138ed45db3f8eeefe: Status 404 returned error can't find the container with id 0f1ca4f224834df51ba490d0f5acd1f8e34ffb0f0c4a03a138ed45db3f8eeefe Jul 21 00:51:36.722001 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.721955 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:51:36.722089 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.722059 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:51:36.722125 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:36.722103 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:51:37.259183 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:37.259143 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" event={"ID":"4d98c01e-14c5-402a-b38d-f1ddd9a408e4","Type":"ContainerStarted","Data":"7ffad2c3b3287de46b23d59b936fd73ecb15cc2a2049dbb64858439737cd1dc0"} Jul 21 00:51:37.259378 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:37.259241 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" event={"ID":"4d98c01e-14c5-402a-b38d-f1ddd9a408e4","Type":"ContainerStarted","Data":"0f1ca4f224834df51ba490d0f5acd1f8e34ffb0f0c4a03a138ed45db3f8eeefe"} Jul 21 00:51:37.281958 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:37.281902 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" podStartSLOduration=1.281885659 podStartE2EDuration="1.281885659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:51:37.281415788 +0000 UTC m=+2091.496796202" watchObservedRunningTime="2026-07-21 00:51:37.281885659 +0000 UTC m=+2091.497265977" Jul 21 00:51:37.570765 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:37.570729 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:37.575767 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:37.575739 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:38.264667 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:38.264638 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:38.265572 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:38.265549 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:39.254190 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:39.254157 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:39.349870 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:39.349832 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:39.417819 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:39.417785 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:39.719370 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:39.719325 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:40.128971 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:40.128933 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:40.229500 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:40.229462 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:43.811501 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:43.811467 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:45.158774 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:45.158718 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:50.627254 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:50.627210 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:50.727224 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:50.727188 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:51:52.362482 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:52.362441 2580 generic.go:356] "Generic (PLEG): container finished" podID="4d98c01e-14c5-402a-b38d-f1ddd9a408e4" containerID="7ffad2c3b3287de46b23d59b936fd73ecb15cc2a2049dbb64858439737cd1dc0" exitCode=0 Jul 21 00:51:52.362946 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:52.362516 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" event={"ID":"4d98c01e-14c5-402a-b38d-f1ddd9a408e4","Type":"ContainerDied","Data":"7ffad2c3b3287de46b23d59b936fd73ecb15cc2a2049dbb64858439737cd1dc0"} Jul 21 00:51:52.363013 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:52.362996 2580 scope.go:117] "RemoveContainer" containerID="7ffad2c3b3287de46b23d59b936fd73ecb15cc2a2049dbb64858439737cd1dc0" Jul 21 00:51:52.363967 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:52.363940 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:51:52.364051 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:52.364006 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:51:52.364051 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:52.364036 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:51:53.264903 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:53.264866 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:53.369994 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:53.369956 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" event={"ID":"4d98c01e-14c5-402a-b38d-f1ddd9a408e4","Type":"ContainerStarted","Data":"4d366fcf25d66d5287533b13aba5e26caf244246935eca282f461a6e030aec2e"} Jul 21 00:51:53.570560 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:53.570530 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:53.571865 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:53.571836 2580 patch_prober.go:28] interesting pod/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Jul 21 00:51:53.571940 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:53.571888 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" podUID="4d98c01e-14c5-402a-b38d-f1ddd9a408e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Jul 21 00:51:54.570545 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:54.570505 2580 patch_prober.go:28] interesting pod/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Jul 21 00:51:54.570985 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:54.570574 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" podUID="4d98c01e-14c5-402a-b38d-f1ddd9a408e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Jul 21 00:51:55.571519 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:55.571475 2580 patch_prober.go:28] interesting pod/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Jul 21 00:51:55.571952 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:55.571537 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" podUID="4d98c01e-14c5-402a-b38d-f1ddd9a408e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Jul 21 00:51:56.570907 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:56.570870 2580 patch_prober.go:28] interesting pod/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Jul 21 00:51:56.571091 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:56.570932 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" podUID="4d98c01e-14c5-402a-b38d-f1ddd9a408e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Jul 21 00:51:57.574956 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:57.574926 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:57.575351 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:57.575336 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:51:57.575958 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:51:57.575939 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:52:01.485089 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:01.485050 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:06.081318 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:06.081266 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:06.546444 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:06.546306 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:12.119982 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:12.119947 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:12.220030 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:12.219991 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:12.364761 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:12.364722 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd"] Jul 21 00:52:12.364996 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:12.364950 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" podUID="4d98c01e-14c5-402a-b38d-f1ddd9a408e4" containerName="istio-proxy" containerID="cri-o://4d366fcf25d66d5287533b13aba5e26caf244246935eca282f461a6e030aec2e" gracePeriod=30 Jul 21 00:52:12.389755 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:12.389577 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:12.578013 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:12.577977 2580 patch_prober.go:28] interesting pod/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 00:52:12.578180 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:12.578037 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" podUID="4d98c01e-14c5-402a-b38d-f1ddd9a408e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 00:52:12.635558 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:12.635523 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:12.756830 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:12.756729 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:12.849585 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:12.849520 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:12.977680 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:12.977624 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:17.511288 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.511256 2580 generic.go:356] "Generic (PLEG): container finished" podID="4d98c01e-14c5-402a-b38d-f1ddd9a408e4" containerID="4d366fcf25d66d5287533b13aba5e26caf244246935eca282f461a6e030aec2e" exitCode=0 Jul 21 00:52:17.511758 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.511332 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" event={"ID":"4d98c01e-14c5-402a-b38d-f1ddd9a408e4","Type":"ContainerDied","Data":"4d366fcf25d66d5287533b13aba5e26caf244246935eca282f461a6e030aec2e"} Jul 21 00:52:17.511758 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.511397 2580 scope.go:117] "RemoveContainer" containerID="7ffad2c3b3287de46b23d59b936fd73ecb15cc2a2049dbb64858439737cd1dc0" Jul 21 00:52:17.632254 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.632183 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:52:17.747516 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.747480 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istio-envoy\") pod \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " Jul 21 00:52:17.747516 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.747521 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-credential-socket\") pod \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " Jul 21 00:52:17.747820 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.747573 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istio-data\") pod \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " Jul 21 00:52:17.747820 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.747589 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-workload-certs\") pod \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " Jul 21 00:52:17.747820 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.747645 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-workload-socket\") pod \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " Jul 21 00:52:17.747820 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.747675 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qj9zw\" (UniqueName: \"kubernetes.io/projected/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-kube-api-access-qj9zw\") pod \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " Jul 21 00:52:17.747820 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.747699 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istio-podinfo\") pod \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " Jul 21 00:52:17.747820 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.747725 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istio-token\") pod \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " Jul 21 00:52:17.747820 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.747758 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istiod-ca-cert\") pod \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\" (UID: \"4d98c01e-14c5-402a-b38d-f1ddd9a408e4\") " Jul 21 00:52:17.748149 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.747940 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4d98c01e-14c5-402a-b38d-f1ddd9a408e4" (UID: "4d98c01e-14c5-402a-b38d-f1ddd9a408e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:52:17.748149 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.747955 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4d98c01e-14c5-402a-b38d-f1ddd9a408e4" (UID: "4d98c01e-14c5-402a-b38d-f1ddd9a408e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:52:17.748149 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.747969 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4d98c01e-14c5-402a-b38d-f1ddd9a408e4" (UID: "4d98c01e-14c5-402a-b38d-f1ddd9a408e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:52:17.748291 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.748164 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-credential-socket\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:52:17.748291 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.748186 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-workload-certs\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:52:17.748291 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.748200 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-workload-socket\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:52:17.748291 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.748271 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4d98c01e-14c5-402a-b38d-f1ddd9a408e4" (UID: "4d98c01e-14c5-402a-b38d-f1ddd9a408e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:52:17.748524 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.748500 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istio-data" (OuterVolumeSpecName: "istio-data") pod "4d98c01e-14c5-402a-b38d-f1ddd9a408e4" (UID: "4d98c01e-14c5-402a-b38d-f1ddd9a408e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:52:17.750192 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.750160 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4d98c01e-14c5-402a-b38d-f1ddd9a408e4" (UID: "4d98c01e-14c5-402a-b38d-f1ddd9a408e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 00:52:17.750296 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.750190 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-kube-api-access-qj9zw" (OuterVolumeSpecName: "kube-api-access-qj9zw") pod "4d98c01e-14c5-402a-b38d-f1ddd9a408e4" (UID: "4d98c01e-14c5-402a-b38d-f1ddd9a408e4"). InnerVolumeSpecName "kube-api-access-qj9zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:52:17.750467 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.750445 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istio-token" (OuterVolumeSpecName: "istio-token") pod "4d98c01e-14c5-402a-b38d-f1ddd9a408e4" (UID: "4d98c01e-14c5-402a-b38d-f1ddd9a408e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:52:17.750467 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.750449 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4d98c01e-14c5-402a-b38d-f1ddd9a408e4" (UID: "4d98c01e-14c5-402a-b38d-f1ddd9a408e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:52:17.849269 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.849228 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istio-envoy\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:52:17.849444 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.849285 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istio-data\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:52:17.849444 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.849295 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj9zw\" (UniqueName: \"kubernetes.io/projected/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-kube-api-access-qj9zw\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:52:17.849444 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.849304 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istio-podinfo\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:52:17.849444 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.849312 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istio-token\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:52:17.849444 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:17.849320 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d98c01e-14c5-402a-b38d-f1ddd9a408e4-istiod-ca-cert\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:52:18.517700 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:18.517663 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" event={"ID":"4d98c01e-14c5-402a-b38d-f1ddd9a408e4","Type":"ContainerDied","Data":"0f1ca4f224834df51ba490d0f5acd1f8e34ffb0f0c4a03a138ed45db3f8eeefe"} Jul 21 00:52:18.517700 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:18.517698 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd" Jul 21 00:52:18.518232 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:18.517722 2580 scope.go:117] "RemoveContainer" containerID="4d366fcf25d66d5287533b13aba5e26caf244246935eca282f461a6e030aec2e" Jul 21 00:52:18.536758 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:18.536725 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd"] Jul 21 00:52:18.540246 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:18.540213 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8c05e2d2-openshift-default-7c69bc6b7f-mldfd"] Jul 21 00:52:18.732937 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:18.732845 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:18.834417 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:18.834389 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:18.955237 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:18.955200 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:19.172191 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:19.172156 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:20.297411 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:20.297379 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d98c01e-14c5-402a-b38d-f1ddd9a408e4" path="/var/lib/kubelet/pods/4d98c01e-14c5-402a-b38d-f1ddd9a408e4/volumes" Jul 21 00:52:25.291227 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:25.291196 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:25.357922 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:25.357893 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:32.035365 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.035325 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:32.335113 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.335075 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g"] Jul 21 00:52:32.335490 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.335478 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d98c01e-14c5-402a-b38d-f1ddd9a408e4" containerName="istio-proxy" Jul 21 00:52:32.335546 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.335492 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d98c01e-14c5-402a-b38d-f1ddd9a408e4" containerName="istio-proxy" Jul 21 00:52:32.335588 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.335569 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d9186ec-24ac-46c3-a214-d8b9c8025672" containerName="istio-proxy" Jul 21 00:52:32.335588 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.335578 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d98c01e-14c5-402a-b38d-f1ddd9a408e4" containerName="istio-proxy" Jul 21 00:52:32.340054 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.340030 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.342931 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.342901 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c6467f0b-openshift-default-dockercfg-fcqm2\"" Jul 21 00:52:32.351593 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.351561 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g"] Jul 21 00:52:32.360901 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.360870 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:32.481819 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.481779 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d40eb48d-89dd-453d-a26c-0ab8811f5939-istio-token\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.481819 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.481818 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-istio-envoy\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.482055 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.481880 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d40eb48d-89dd-453d-a26c-0ab8811f5939-istiod-ca-cert\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.482055 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.481917 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-workload-certs\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.482055 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.481958 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qtqn\" (UniqueName: \"kubernetes.io/projected/d40eb48d-89dd-453d-a26c-0ab8811f5939-kube-api-access-2qtqn\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.482055 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.482009 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-istio-data\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.482055 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.482026 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d40eb48d-89dd-453d-a26c-0ab8811f5939-istio-podinfo\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.482055 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.482050 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-workload-socket\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.482251 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.482067 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-credential-socket\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.590387 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.583184 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d40eb48d-89dd-453d-a26c-0ab8811f5939-istio-token\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.590387 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.583272 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-istio-envoy\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.590387 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.583317 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d40eb48d-89dd-453d-a26c-0ab8811f5939-istiod-ca-cert\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.590387 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.583382 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-workload-certs\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.590387 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.583449 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qtqn\" (UniqueName: \"kubernetes.io/projected/d40eb48d-89dd-453d-a26c-0ab8811f5939-kube-api-access-2qtqn\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.590387 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.583511 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-istio-data\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.590387 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.583546 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d40eb48d-89dd-453d-a26c-0ab8811f5939-istio-podinfo\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.590387 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.583591 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-workload-socket\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.590387 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.583648 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-credential-socket\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.590387 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.584149 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-credential-socket\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.590387 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.587414 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-istio-envoy\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.590387 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.588099 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d40eb48d-89dd-453d-a26c-0ab8811f5939-istiod-ca-cert\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.590387 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.588341 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-workload-certs\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.590387 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.588987 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-istio-data\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.591228 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.590398 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-workload-socket\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.592313 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.592289 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d40eb48d-89dd-453d-a26c-0ab8811f5939-istio-podinfo\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.595263 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.595231 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:32.598447 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.598424 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d40eb48d-89dd-453d-a26c-0ab8811f5939-istio-token\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.600533 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.600512 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qtqn\" (UniqueName: \"kubernetes.io/projected/d40eb48d-89dd-453d-a26c-0ab8811f5939-kube-api-access-2qtqn\") pod \"e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.651992 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.651957 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:32.801543 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.801502 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g"] Jul 21 00:52:32.803011 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:52:32.802974 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40eb48d_89dd_453d_a26c_0ab8811f5939.slice/crio-3b3bf7eb1534951ca71a6f99c3cc8e52ed92059fc072f46bb65268fb3e2bcc5a WatchSource:0}: Error finding container 3b3bf7eb1534951ca71a6f99c3cc8e52ed92059fc072f46bb65268fb3e2bcc5a: Status 404 returned error can't find the container with id 3b3bf7eb1534951ca71a6f99c3cc8e52ed92059fc072f46bb65268fb3e2bcc5a Jul 21 00:52:32.805119 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.805086 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:52:32.805200 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.805153 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:52:32.805200 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:32.805182 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:52:33.608650 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:33.608615 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" event={"ID":"d40eb48d-89dd-453d-a26c-0ab8811f5939","Type":"ContainerStarted","Data":"3e85aad5e1fb8dd64f7f63a0ee73678dea097b7d4a6d6a32878c966b4f790e7d"} Jul 21 00:52:33.609054 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:33.608668 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" event={"ID":"d40eb48d-89dd-453d-a26c-0ab8811f5939","Type":"ContainerStarted","Data":"3b3bf7eb1534951ca71a6f99c3cc8e52ed92059fc072f46bb65268fb3e2bcc5a"} Jul 21 00:52:33.632372 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:33.632315 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" podStartSLOduration=1.632299602 podStartE2EDuration="1.632299602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:52:33.629815315 +0000 UTC m=+2147.845195672" watchObservedRunningTime="2026-07-21 00:52:33.632299602 +0000 UTC m=+2147.847680007" Jul 21 00:52:33.653086 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:33.653052 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:33.658047 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:33.658020 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:34.615036 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:34.615004 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:34.616122 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:34.616101 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:35.667970 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:35.667931 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:35.754894 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:35.754835 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:35.827122 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:35.827069 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:35.927383 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:35.927294 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:36.345880 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:36.345851 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:36.446979 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:36.446933 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:40.133760 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:40.133709 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:48.697868 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:48.697805 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:52:48.701343 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:48.701316 2580 generic.go:356] "Generic (PLEG): container finished" podID="d40eb48d-89dd-453d-a26c-0ab8811f5939" containerID="3e85aad5e1fb8dd64f7f63a0ee73678dea097b7d4a6d6a32878c966b4f790e7d" exitCode=0 Jul 21 00:52:48.701496 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:48.701396 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" event={"ID":"d40eb48d-89dd-453d-a26c-0ab8811f5939","Type":"ContainerDied","Data":"3e85aad5e1fb8dd64f7f63a0ee73678dea097b7d4a6d6a32878c966b4f790e7d"} Jul 21 00:52:48.701906 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:48.701888 2580 scope.go:117] "RemoveContainer" containerID="3e85aad5e1fb8dd64f7f63a0ee73678dea097b7d4a6d6a32878c966b4f790e7d" Jul 21 00:52:48.702959 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:48.702933 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:52:48.703027 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:48.703006 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:52:48.703067 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:48.703037 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:52:49.615117 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:49.615070 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:49.713580 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:49.713544 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" event={"ID":"d40eb48d-89dd-453d-a26c-0ab8811f5939","Type":"ContainerStarted","Data":"6418e5b8ec77f1bb2994de9a6d111b9b2846a5dfb22a747a476a97de6668fd8e"} Jul 21 00:52:50.652213 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:50.652173 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:50.653545 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:50.653519 2580 patch_prober.go:28] interesting pod/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 21 00:52:50.653693 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:50.653571 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" podUID="d40eb48d-89dd-453d-a26c-0ab8811f5939" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 21 00:52:51.652973 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:51.652935 2580 patch_prober.go:28] interesting pod/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 21 00:52:51.653388 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:51.653042 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" podUID="d40eb48d-89dd-453d-a26c-0ab8811f5939" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 21 00:52:52.652785 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:52.652749 2580 patch_prober.go:28] interesting pod/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 21 00:52:52.653047 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:52.652807 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" podUID="d40eb48d-89dd-453d-a26c-0ab8811f5939" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 21 00:52:53.656095 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:53.656062 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:53.656548 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:53.656496 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:53.657350 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:53.657331 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:52:58.505685 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:52:58.505640 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:53:24.722566 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:53:24.722506 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:53:29.252901 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:53:29.252860 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:53:31.627611 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:53:31.627578 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:53:36.655328 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:53:36.655288 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:53:48.127950 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:53:48.127918 2580 scope.go:117] "RemoveContainer" containerID="dee3f07649220b0d82c8da113500270e1cfb6a9dec99131561d9b778cb3bf8a9" Jul 21 00:54:13.240720 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:13.240623 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:54:14.715335 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:14.715295 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:54:20.235767 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:20.235720 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:54:20.390048 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:20.390010 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:54:20.536683 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:20.536557 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g"] Jul 21 00:54:20.536868 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:20.536841 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" podUID="d40eb48d-89dd-453d-a26c-0ab8811f5939" containerName="istio-proxy" containerID="cri-o://6418e5b8ec77f1bb2994de9a6d111b9b2846a5dfb22a747a476a97de6668fd8e" gracePeriod=30 Jul 21 00:54:20.559668 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:20.559630 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:54:20.794848 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:20.794762 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:54:21.917594 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:21.917554 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:54:22.022176 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:22.022142 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:54:22.117801 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:22.117772 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:54:22.922358 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:22.922323 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:54:23.020154 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:23.020121 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:54:23.122629 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:23.122572 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:54:23.658836 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:23.658792 2580 patch_prober.go:28] interesting pod/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 00:54:23.659015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:23.658866 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" podUID="d40eb48d-89dd-453d-a26c-0ab8811f5939" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 00:54:24.592025 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:24.591992 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:54:25.793194 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.793162 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:54:25.867894 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.867851 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-workload-certs\") pod \"d40eb48d-89dd-453d-a26c-0ab8811f5939\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " Jul 21 00:54:25.867894 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.867901 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d40eb48d-89dd-453d-a26c-0ab8811f5939-istio-token\") pod \"d40eb48d-89dd-453d-a26c-0ab8811f5939\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " Jul 21 00:54:25.868155 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.867924 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-istio-data\") pod \"d40eb48d-89dd-453d-a26c-0ab8811f5939\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " Jul 21 00:54:25.868155 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.867955 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d40eb48d-89dd-453d-a26c-0ab8811f5939-istiod-ca-cert\") pod \"d40eb48d-89dd-453d-a26c-0ab8811f5939\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " Jul 21 00:54:25.868155 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.867980 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qtqn\" (UniqueName: \"kubernetes.io/projected/d40eb48d-89dd-453d-a26c-0ab8811f5939-kube-api-access-2qtqn\") pod \"d40eb48d-89dd-453d-a26c-0ab8811f5939\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " Jul 21 00:54:25.868155 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.868005 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-workload-socket\") pod \"d40eb48d-89dd-453d-a26c-0ab8811f5939\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " Jul 21 00:54:25.868155 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.868040 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-istio-envoy\") pod \"d40eb48d-89dd-453d-a26c-0ab8811f5939\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " Jul 21 00:54:25.868155 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.868084 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d40eb48d-89dd-453d-a26c-0ab8811f5939-istio-podinfo\") pod \"d40eb48d-89dd-453d-a26c-0ab8811f5939\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " Jul 21 00:54:25.868155 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.868142 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-credential-socket\") pod \"d40eb48d-89dd-453d-a26c-0ab8811f5939\" (UID: \"d40eb48d-89dd-453d-a26c-0ab8811f5939\") " Jul 21 00:54:25.868497 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.868182 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d40eb48d-89dd-453d-a26c-0ab8811f5939" (UID: "d40eb48d-89dd-453d-a26c-0ab8811f5939"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:54:25.868497 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.868417 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-workload-certs\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:54:25.868497 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.868410 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d40eb48d-89dd-453d-a26c-0ab8811f5939-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d40eb48d-89dd-453d-a26c-0ab8811f5939" (UID: "d40eb48d-89dd-453d-a26c-0ab8811f5939"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:54:25.868688 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.868551 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d40eb48d-89dd-453d-a26c-0ab8811f5939" (UID: "d40eb48d-89dd-453d-a26c-0ab8811f5939"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:54:25.868756 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.868680 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d40eb48d-89dd-453d-a26c-0ab8811f5939" (UID: "d40eb48d-89dd-453d-a26c-0ab8811f5939"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:54:25.869263 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.869225 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-istio-data" (OuterVolumeSpecName: "istio-data") pod "d40eb48d-89dd-453d-a26c-0ab8811f5939" (UID: "d40eb48d-89dd-453d-a26c-0ab8811f5939"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:54:25.870360 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.870327 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40eb48d-89dd-453d-a26c-0ab8811f5939-istio-token" (OuterVolumeSpecName: "istio-token") pod "d40eb48d-89dd-453d-a26c-0ab8811f5939" (UID: "d40eb48d-89dd-453d-a26c-0ab8811f5939"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:54:25.870467 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.870414 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d40eb48d-89dd-453d-a26c-0ab8811f5939-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d40eb48d-89dd-453d-a26c-0ab8811f5939" (UID: "d40eb48d-89dd-453d-a26c-0ab8811f5939"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 00:54:25.870467 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.870422 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40eb48d-89dd-453d-a26c-0ab8811f5939-kube-api-access-2qtqn" (OuterVolumeSpecName: "kube-api-access-2qtqn") pod "d40eb48d-89dd-453d-a26c-0ab8811f5939" (UID: "d40eb48d-89dd-453d-a26c-0ab8811f5939"). InnerVolumeSpecName "kube-api-access-2qtqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:54:25.870551 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.870478 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d40eb48d-89dd-453d-a26c-0ab8811f5939" (UID: "d40eb48d-89dd-453d-a26c-0ab8811f5939"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:54:25.968936 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.968904 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d40eb48d-89dd-453d-a26c-0ab8811f5939-istio-token\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:54:25.968936 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.968941 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-istio-data\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:54:25.969145 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.968949 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d40eb48d-89dd-453d-a26c-0ab8811f5939-istiod-ca-cert\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:54:25.969145 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.968958 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qtqn\" (UniqueName: \"kubernetes.io/projected/d40eb48d-89dd-453d-a26c-0ab8811f5939-kube-api-access-2qtqn\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:54:25.969145 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.968968 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-workload-socket\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:54:25.969145 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.968976 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-istio-envoy\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:54:25.969145 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.968984 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d40eb48d-89dd-453d-a26c-0ab8811f5939-istio-podinfo\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:54:25.969145 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:25.968991 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d40eb48d-89dd-453d-a26c-0ab8811f5939-credential-socket\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:54:26.272936 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:26.272841 2580 generic.go:356] "Generic (PLEG): container finished" podID="d40eb48d-89dd-453d-a26c-0ab8811f5939" containerID="6418e5b8ec77f1bb2994de9a6d111b9b2846a5dfb22a747a476a97de6668fd8e" exitCode=0 Jul 21 00:54:26.272936 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:26.272891 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" Jul 21 00:54:26.272936 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:26.272924 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" event={"ID":"d40eb48d-89dd-453d-a26c-0ab8811f5939","Type":"ContainerDied","Data":"6418e5b8ec77f1bb2994de9a6d111b9b2846a5dfb22a747a476a97de6668fd8e"} Jul 21 00:54:26.273206 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:26.272976 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g" event={"ID":"d40eb48d-89dd-453d-a26c-0ab8811f5939","Type":"ContainerDied","Data":"3b3bf7eb1534951ca71a6f99c3cc8e52ed92059fc072f46bb65268fb3e2bcc5a"} Jul 21 00:54:26.273206 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:26.272998 2580 scope.go:117] "RemoveContainer" containerID="6418e5b8ec77f1bb2994de9a6d111b9b2846a5dfb22a747a476a97de6668fd8e" Jul 21 00:54:26.283313 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:26.283290 2580 scope.go:117] "RemoveContainer" containerID="3e85aad5e1fb8dd64f7f63a0ee73678dea097b7d4a6d6a32878c966b4f790e7d" Jul 21 00:54:26.292988 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:26.292970 2580 scope.go:117] "RemoveContainer" containerID="6418e5b8ec77f1bb2994de9a6d111b9b2846a5dfb22a747a476a97de6668fd8e" Jul 21 00:54:26.293388 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:54:26.293359 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6418e5b8ec77f1bb2994de9a6d111b9b2846a5dfb22a747a476a97de6668fd8e\": container with ID starting with 6418e5b8ec77f1bb2994de9a6d111b9b2846a5dfb22a747a476a97de6668fd8e not found: ID does not exist" containerID="6418e5b8ec77f1bb2994de9a6d111b9b2846a5dfb22a747a476a97de6668fd8e" Jul 21 00:54:26.293498 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:26.293400 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6418e5b8ec77f1bb2994de9a6d111b9b2846a5dfb22a747a476a97de6668fd8e"} err="failed to get container status \"6418e5b8ec77f1bb2994de9a6d111b9b2846a5dfb22a747a476a97de6668fd8e\": rpc error: code = NotFound desc = could not find container \"6418e5b8ec77f1bb2994de9a6d111b9b2846a5dfb22a747a476a97de6668fd8e\": container with ID starting with 6418e5b8ec77f1bb2994de9a6d111b9b2846a5dfb22a747a476a97de6668fd8e not found: ID does not exist" Jul 21 00:54:26.293498 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:26.293426 2580 scope.go:117] "RemoveContainer" containerID="3e85aad5e1fb8dd64f7f63a0ee73678dea097b7d4a6d6a32878c966b4f790e7d" Jul 21 00:54:26.293969 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:54:26.293942 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e85aad5e1fb8dd64f7f63a0ee73678dea097b7d4a6d6a32878c966b4f790e7d\": container with ID starting with 3e85aad5e1fb8dd64f7f63a0ee73678dea097b7d4a6d6a32878c966b4f790e7d not found: ID does not exist" containerID="3e85aad5e1fb8dd64f7f63a0ee73678dea097b7d4a6d6a32878c966b4f790e7d" Jul 21 00:54:26.294071 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:26.293997 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e85aad5e1fb8dd64f7f63a0ee73678dea097b7d4a6d6a32878c966b4f790e7d"} err="failed to get container status \"3e85aad5e1fb8dd64f7f63a0ee73678dea097b7d4a6d6a32878c966b4f790e7d\": rpc error: code = NotFound desc = could not find container \"3e85aad5e1fb8dd64f7f63a0ee73678dea097b7d4a6d6a32878c966b4f790e7d\": container with ID starting with 3e85aad5e1fb8dd64f7f63a0ee73678dea097b7d4a6d6a32878c966b4f790e7d not found: ID does not exist" Jul 21 00:54:26.303574 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:26.303545 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g"] Jul 21 00:54:26.303574 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:26.303581 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c6467f0b-openshift-default-65d77695d7-cj96g"] Jul 21 00:54:28.303127 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:28.303090 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d40eb48d-89dd-453d-a26c-0ab8811f5939" path="/var/lib/kubelet/pods/d40eb48d-89dd-453d-a26c-0ab8811f5939/volumes" Jul 21 00:54:30.415853 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:30.415821 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:54:30.515936 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:30.515904 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:54:31.147267 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:31.147227 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:54:34.501716 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:34.501672 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:54:34.601775 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:34.601743 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:54:37.740974 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:37.740935 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:54:37.845163 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:37.845130 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:54:37.943295 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:37.943266 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:54:48.529636 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:48.529543 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:54:48.910189 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:54:48.910153 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:09.866684 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:09.866640 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:16.144015 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.143980 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:16.244051 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.244018 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:16.520532 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.520454 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:16.814853 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.814824 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7"] Jul 21 00:55:16.815234 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.815223 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d40eb48d-89dd-453d-a26c-0ab8811f5939" containerName="istio-proxy" Jul 21 00:55:16.815282 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.815236 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d40eb48d-89dd-453d-a26c-0ab8811f5939" containerName="istio-proxy" Jul 21 00:55:16.815282 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.815256 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d40eb48d-89dd-453d-a26c-0ab8811f5939" containerName="istio-proxy" Jul 21 00:55:16.815282 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.815262 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d40eb48d-89dd-453d-a26c-0ab8811f5939" containerName="istio-proxy" Jul 21 00:55:16.815282 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.815270 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d98c01e-14c5-402a-b38d-f1ddd9a408e4" containerName="istio-proxy" Jul 21 00:55:16.815282 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.815275 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d98c01e-14c5-402a-b38d-f1ddd9a408e4" containerName="istio-proxy" Jul 21 00:55:16.815461 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.815331 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d40eb48d-89dd-453d-a26c-0ab8811f5939" containerName="istio-proxy" Jul 21 00:55:16.815461 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.815341 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d98c01e-14c5-402a-b38d-f1ddd9a408e4" containerName="istio-proxy" Jul 21 00:55:16.815461 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.815347 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d40eb48d-89dd-453d-a26c-0ab8811f5939" containerName="istio-proxy" Jul 21 00:55:16.818545 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.818527 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:16.820897 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.820877 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1c3d09-openshift-default-dockercfg-pjlbg\"" Jul 21 00:55:16.831371 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.831335 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7"] Jul 21 00:55:16.849025 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.849000 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:16.955495 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.955460 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-istio-data\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:16.955698 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.955510 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65509139-5b21-42bd-ba31-bb84b7fbb77f-istiod-ca-cert\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:16.955698 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.955530 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kf6x\" (UniqueName: \"kubernetes.io/projected/65509139-5b21-42bd-ba31-bb84b7fbb77f-kube-api-access-5kf6x\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:16.955698 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.955590 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-istio-envoy\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:16.955698 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.955666 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-workload-socket\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:16.955843 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.955701 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65509139-5b21-42bd-ba31-bb84b7fbb77f-istio-podinfo\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:16.955843 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.955745 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-credential-socket\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:16.955843 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.955783 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-workload-certs\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:16.955843 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:16.955804 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65509139-5b21-42bd-ba31-bb84b7fbb77f-istio-token\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:17.056766 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.056733 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-workload-socket\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:17.056766 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.056770 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65509139-5b21-42bd-ba31-bb84b7fbb77f-istio-podinfo\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:17.056985 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.056885 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-credential-socket\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:17.056985 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.056939 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-workload-certs\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:17.056985 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.056965 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65509139-5b21-42bd-ba31-bb84b7fbb77f-istio-token\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:17.057105 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.057030 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-istio-data\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:17.057105 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.057078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65509139-5b21-42bd-ba31-bb84b7fbb77f-istiod-ca-cert\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:17.057105 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.057101 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kf6x\" (UniqueName: \"kubernetes.io/projected/65509139-5b21-42bd-ba31-bb84b7fbb77f-kube-api-access-5kf6x\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:17.057259 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.057148 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-istio-envoy\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:17.057259 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.057198 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-credential-socket\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:17.057259 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.057231 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-workload-socket\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:17.057416 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.057256 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-workload-certs\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:17.057416 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.057371 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-istio-data\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:17.057822 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.057803 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65509139-5b21-42bd-ba31-bb84b7fbb77f-istiod-ca-cert\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:17.059361 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.059344 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-istio-envoy\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:17.059441 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.059428 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65509139-5b21-42bd-ba31-bb84b7fbb77f-istio-podinfo\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:17.092340 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.092264 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kf6x\" (UniqueName: \"kubernetes.io/projected/65509139-5b21-42bd-ba31-bb84b7fbb77f-kube-api-access-5kf6x\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:17.092503 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.092404 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65509139-5b21-42bd-ba31-bb84b7fbb77f-istio-token\") pod \"e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:17.099223 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.099194 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:17.128063 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.128037 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:17.269411 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.269383 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7"] Jul 21 00:55:17.270676 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:55:17.270653 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65509139_5b21_42bd_ba31_bb84b7fbb77f.slice/crio-4536b61ed1b5bc59a966621a51b4ef0304a23bb94aefba3dc5ae159d60f8a86a WatchSource:0}: Error finding container 4536b61ed1b5bc59a966621a51b4ef0304a23bb94aefba3dc5ae159d60f8a86a: Status 404 returned error can't find the container with id 4536b61ed1b5bc59a966621a51b4ef0304a23bb94aefba3dc5ae159d60f8a86a Jul 21 00:55:17.272999 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.272967 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:55:17.273064 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.273030 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:55:17.273064 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.273056 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:55:17.560536 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.560500 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" event={"ID":"65509139-5b21-42bd-ba31-bb84b7fbb77f","Type":"ContainerStarted","Data":"8786b9b57ba83073ba14b4f29bcf7a24cc85fff0bfb89b2df4675c5879ac4636"} Jul 21 00:55:17.560751 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.560559 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" event={"ID":"65509139-5b21-42bd-ba31-bb84b7fbb77f","Type":"ContainerStarted","Data":"4536b61ed1b5bc59a966621a51b4ef0304a23bb94aefba3dc5ae159d60f8a86a"} Jul 21 00:55:17.605575 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:17.605515 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" podStartSLOduration=1.60549719 podStartE2EDuration="1.60549719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:55:17.602952525 +0000 UTC m=+2311.818332852" watchObservedRunningTime="2026-07-21 00:55:17.60549719 +0000 UTC m=+2311.820877515" Jul 21 00:55:18.128746 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:18.128716 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:18.133304 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:18.133280 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:18.565673 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:18.565648 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:18.566473 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:18.566454 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:19.694133 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:19.694103 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:19.793475 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:19.793442 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:19.863280 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:19.863245 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:19.989583 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:19.989495 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:20.464995 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:20.464966 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:20.569028 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:20.568999 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:24.133336 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.133304 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:24.472623 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.472521 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:24.748897 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.748805 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn"] Jul 21 00:55:24.754785 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.754751 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.762741 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.762686 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ee9ef4-openshift-default-dockercfg-9qpk9\"" Jul 21 00:55:24.766462 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.766434 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn"] Jul 21 00:55:24.792501 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.792462 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:24.832846 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.832813 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b77d75-567a-4112-b28a-15a134cd9bd5-istio-podinfo\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.832846 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.832852 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-istio-data\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.833087 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.832934 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-workload-socket\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.833087 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.832972 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b77d75-567a-4112-b28a-15a134cd9bd5-istio-token\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.833087 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.832989 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlzcg\" (UniqueName: \"kubernetes.io/projected/59b77d75-567a-4112-b28a-15a134cd9bd5-kube-api-access-qlzcg\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.833087 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.833013 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b77d75-567a-4112-b28a-15a134cd9bd5-istiod-ca-cert\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.833087 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.833079 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-credential-socket\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.833262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.833102 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-workload-certs\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.833262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.833153 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-istio-envoy\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.933884 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.933850 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b77d75-567a-4112-b28a-15a134cd9bd5-istio-token\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.933884 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.933886 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlzcg\" (UniqueName: \"kubernetes.io/projected/59b77d75-567a-4112-b28a-15a134cd9bd5-kube-api-access-qlzcg\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.934115 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.933920 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b77d75-567a-4112-b28a-15a134cd9bd5-istiod-ca-cert\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.934115 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.933966 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-credential-socket\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.934115 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.933983 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-workload-certs\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.934115 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.934105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-istio-envoy\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.934316 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.934216 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b77d75-567a-4112-b28a-15a134cd9bd5-istio-podinfo\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.934316 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.934260 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-istio-data\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.934428 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.934335 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-credential-socket\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.934428 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.934337 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-workload-socket\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.934510 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.934419 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-workload-certs\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.934592 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.934572 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-workload-socket\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.934683 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.934662 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-istio-data\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.934731 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.934717 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b77d75-567a-4112-b28a-15a134cd9bd5-istiod-ca-cert\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.936621 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.936581 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-istio-envoy\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.936902 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.936885 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b77d75-567a-4112-b28a-15a134cd9bd5-istio-podinfo\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.941939 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.941920 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b77d75-567a-4112-b28a-15a134cd9bd5-istio-token\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.942096 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.942075 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlzcg\" (UniqueName: \"kubernetes.io/projected/59b77d75-567a-4112-b28a-15a134cd9bd5-kube-api-access-qlzcg\") pod \"e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:24.971736 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:24.971710 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:25.070795 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:25.070764 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:25.217933 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:25.217904 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn"] Jul 21 00:55:25.219248 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:55:25.219217 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b77d75_567a_4112_b28a_15a134cd9bd5.slice/crio-d2820f80d61d36e31249c010d9dc5bcfc5d0b8671e1987db2e0b4c93a9206f03 WatchSource:0}: Error finding container d2820f80d61d36e31249c010d9dc5bcfc5d0b8671e1987db2e0b4c93a9206f03: Status 404 returned error can't find the container with id d2820f80d61d36e31249c010d9dc5bcfc5d0b8671e1987db2e0b4c93a9206f03 Jul 21 00:55:25.221394 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:25.221361 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:55:25.221476 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:25.221427 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:55:25.221476 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:25.221456 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:55:25.608164 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:25.608120 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" event={"ID":"59b77d75-567a-4112-b28a-15a134cd9bd5","Type":"ContainerStarted","Data":"92a0eda9a8497d6a0b3e7e1e0f4ef7c8ab27063e686feea17730e0a2e0edb044"} Jul 21 00:55:25.608374 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:25.608183 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" event={"ID":"59b77d75-567a-4112-b28a-15a134cd9bd5","Type":"ContainerStarted","Data":"d2820f80d61d36e31249c010d9dc5bcfc5d0b8671e1987db2e0b4c93a9206f03"} Jul 21 00:55:25.631331 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:25.631280 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" podStartSLOduration=1.631264017 podStartE2EDuration="1.631264017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:55:25.628115053 +0000 UTC m=+2319.843495381" watchObservedRunningTime="2026-07-21 00:55:25.631264017 +0000 UTC m=+2319.846644341" Jul 21 00:55:26.071262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:26.071225 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:26.076285 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:26.076260 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:26.616152 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:26.616121 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:26.617428 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:26.617392 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:27.607060 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:27.607009 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:27.706178 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:27.706141 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:27.931141 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:27.931062 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:28.139706 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:28.139648 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:28.452120 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:28.452085 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:28.553098 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:28.553070 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:32.107458 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:32.107383 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:33.567500 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:33.567464 2580 patch_prober.go:28] interesting pod/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 21 00:55:33.567929 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:33.567530 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" podUID="65509139-5b21-42bd-ba31-bb84b7fbb77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 21 00:55:33.660283 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:33.660190 2580 generic.go:356] "Generic (PLEG): container finished" podID="65509139-5b21-42bd-ba31-bb84b7fbb77f" containerID="8786b9b57ba83073ba14b4f29bcf7a24cc85fff0bfb89b2df4675c5879ac4636" exitCode=0 Jul 21 00:55:33.660283 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:33.660269 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" event={"ID":"65509139-5b21-42bd-ba31-bb84b7fbb77f","Type":"ContainerDied","Data":"8786b9b57ba83073ba14b4f29bcf7a24cc85fff0bfb89b2df4675c5879ac4636"} Jul 21 00:55:33.660756 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:33.660740 2580 scope.go:117] "RemoveContainer" containerID="8786b9b57ba83073ba14b4f29bcf7a24cc85fff0bfb89b2df4675c5879ac4636" Jul 21 00:55:33.661876 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:33.661844 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:55:33.661959 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:33.661935 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:55:33.662000 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:33.661980 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:55:34.667658 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:34.667626 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" event={"ID":"65509139-5b21-42bd-ba31-bb84b7fbb77f","Type":"ContainerStarted","Data":"721b56840b6e94d978fcd05dd0436c371856ffd4792b110adf43716933d8be0b"} Jul 21 00:55:35.130051 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:35.129566 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:35.130051 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:35.129735 2580 patch_prober.go:28] interesting pod/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 21 00:55:35.130051 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:35.129781 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" podUID="65509139-5b21-42bd-ba31-bb84b7fbb77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 21 00:55:36.129495 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:36.129450 2580 patch_prober.go:28] interesting pod/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 21 00:55:36.129942 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:36.129513 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" podUID="65509139-5b21-42bd-ba31-bb84b7fbb77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 21 00:55:37.128727 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:37.128628 2580 patch_prober.go:28] interesting pod/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 21 00:55:37.128727 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:37.128699 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" podUID="65509139-5b21-42bd-ba31-bb84b7fbb77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 21 00:55:37.543653 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:37.543545 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:38.129501 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:38.129464 2580 patch_prober.go:28] interesting pod/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 21 00:55:38.129715 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:38.129523 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" podUID="65509139-5b21-42bd-ba31-bb84b7fbb77f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 21 00:55:39.132625 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:39.132576 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:39.133022 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:39.132931 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:39.133523 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:39.133497 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:55:39.699941 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:39.699909 2580 generic.go:356] "Generic (PLEG): container finished" podID="59b77d75-567a-4112-b28a-15a134cd9bd5" containerID="92a0eda9a8497d6a0b3e7e1e0f4ef7c8ab27063e686feea17730e0a2e0edb044" exitCode=0 Jul 21 00:55:39.700155 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:39.699980 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" event={"ID":"59b77d75-567a-4112-b28a-15a134cd9bd5","Type":"ContainerDied","Data":"92a0eda9a8497d6a0b3e7e1e0f4ef7c8ab27063e686feea17730e0a2e0edb044"} Jul 21 00:55:39.700519 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:39.700500 2580 scope.go:117] "RemoveContainer" containerID="92a0eda9a8497d6a0b3e7e1e0f4ef7c8ab27063e686feea17730e0a2e0edb044" Jul 21 00:55:39.701493 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:39.701468 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:55:39.701556 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:39.701530 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:55:39.701594 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:39.701574 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:55:40.712701 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:40.712662 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" event={"ID":"59b77d75-567a-4112-b28a-15a134cd9bd5","Type":"ContainerStarted","Data":"7ec0c94720d2d482e22de44a98a23522a21a4ffb869562015d2f4b036314cb57"} Jul 21 00:55:41.071172 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:41.071139 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:41.072394 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:41.072369 2580 patch_prober.go:28] interesting pod/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Jul 21 00:55:41.072523 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:41.072418 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" podUID="59b77d75-567a-4112-b28a-15a134cd9bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Jul 21 00:55:41.616234 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:41.616187 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:42.071859 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:42.071822 2580 patch_prober.go:28] interesting pod/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Jul 21 00:55:42.072247 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:42.071880 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" podUID="59b77d75-567a-4112-b28a-15a134cd9bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Jul 21 00:55:42.328572 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:42.328485 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:42.430852 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:42.430813 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:43.071260 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:43.071226 2580 patch_prober.go:28] interesting pod/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Jul 21 00:55:43.071458 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:43.071287 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" podUID="59b77d75-567a-4112-b28a-15a134cd9bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Jul 21 00:55:44.071876 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:44.071844 2580 patch_prober.go:28] interesting pod/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Jul 21 00:55:44.072262 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:44.071894 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" podUID="59b77d75-567a-4112-b28a-15a134cd9bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Jul 21 00:55:45.076042 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:45.076012 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:45.076897 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:45.076876 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:55:49.335234 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:49.335204 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:49.437366 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:49.437329 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:55.544656 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.544620 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4"] Jul 21 00:55:55.548184 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.548160 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:55:55.550575 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.550554 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ee9ef4\"/\"openshift-service-ca.crt\"" Jul 21 00:55:55.551505 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.551475 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ee9ef4\"/\"kube-root-ca.crt\"" Jul 21 00:55:55.551592 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.551531 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ee9ef4\"/\"test-model-ee9ef4-kserve-self-signed-certs\"" Jul 21 00:55:55.551663 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.551531 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ee9ef4\"/\"default-dockercfg-59t54\"" Jul 21 00:55:55.556050 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.556032 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4"] Jul 21 00:55:55.642879 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.642851 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btwck\" (UniqueName: \"kubernetes.io/projected/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-kube-api-access-btwck\") pod \"test-model-ee9ef4-kserve-7b6c75d6d9-499j4\" (UID: \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\") " pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:55:55.643071 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.642910 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-dshm\") pod \"test-model-ee9ef4-kserve-7b6c75d6d9-499j4\" (UID: \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\") " pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:55:55.643071 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.642953 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-tls-certs\") pod \"test-model-ee9ef4-kserve-7b6c75d6d9-499j4\" (UID: \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\") " pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:55:55.643071 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.642972 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-tmp-dir\") pod \"test-model-ee9ef4-kserve-7b6c75d6d9-499j4\" (UID: \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\") " pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:55:55.643071 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.643007 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-home\") pod \"test-model-ee9ef4-kserve-7b6c75d6d9-499j4\" (UID: \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\") " pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:55:55.643226 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.643087 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-model-cache\") pod \"test-model-ee9ef4-kserve-7b6c75d6d9-499j4\" (UID: \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\") " pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:55:55.669871 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.669839 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:55:55.744127 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.744089 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-dshm\") pod \"test-model-ee9ef4-kserve-7b6c75d6d9-499j4\" (UID: \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\") " pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:55:55.744127 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.744132 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-tls-certs\") pod \"test-model-ee9ef4-kserve-7b6c75d6d9-499j4\" (UID: \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\") " pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:55:55.744386 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.744258 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-tmp-dir\") pod \"test-model-ee9ef4-kserve-7b6c75d6d9-499j4\" (UID: \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\") " pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:55:55.744386 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.744334 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-home\") pod \"test-model-ee9ef4-kserve-7b6c75d6d9-499j4\" (UID: \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\") " pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:55:55.744386 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.744374 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-model-cache\") pod \"test-model-ee9ef4-kserve-7b6c75d6d9-499j4\" (UID: \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\") " pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:55:55.744578 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.744461 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btwck\" (UniqueName: \"kubernetes.io/projected/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-kube-api-access-btwck\") pod \"test-model-ee9ef4-kserve-7b6c75d6d9-499j4\" (UID: \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\") " pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:55:55.744683 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.744637 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-tmp-dir\") pod \"test-model-ee9ef4-kserve-7b6c75d6d9-499j4\" (UID: \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\") " pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:55:55.744737 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.744687 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-home\") pod \"test-model-ee9ef4-kserve-7b6c75d6d9-499j4\" (UID: \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\") " pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:55:55.744797 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.744777 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-model-cache\") pod \"test-model-ee9ef4-kserve-7b6c75d6d9-499j4\" (UID: \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\") " pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:55:55.746518 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.746493 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-dshm\") pod \"test-model-ee9ef4-kserve-7b6c75d6d9-499j4\" (UID: \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\") " pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:55:55.746688 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.746671 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-tls-certs\") pod \"test-model-ee9ef4-kserve-7b6c75d6d9-499j4\" (UID: \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\") " pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:55:55.752485 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.752463 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btwck\" (UniqueName: \"kubernetes.io/projected/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-kube-api-access-btwck\") pod \"test-model-ee9ef4-kserve-7b6c75d6d9-499j4\" (UID: \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\") " pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:55:55.858162 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.858127 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:55:55.995453 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:55.995423 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4"] Jul 21 00:55:55.996367 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:55:55.996342 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cf717cf_7850_4039_a7c1_1ebb7d0f4b84.slice/crio-e942eb6d6766bd9262b738722f8a8af6b5dd0ab63f3f185d699900dd91a28a71 WatchSource:0}: Error finding container e942eb6d6766bd9262b738722f8a8af6b5dd0ab63f3f185d699900dd91a28a71: Status 404 returned error can't find the container with id e942eb6d6766bd9262b738722f8a8af6b5dd0ab63f3f185d699900dd91a28a71 Jul 21 00:55:56.812915 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:56.812877 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" event={"ID":"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84","Type":"ContainerStarted","Data":"aa554a11636b3b7dce6d614cb6edb4f5171719d10b51d166d8dacc4474a7a6f7"} Jul 21 00:55:56.812915 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:56.812923 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" event={"ID":"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84","Type":"ContainerStarted","Data":"e942eb6d6766bd9262b738722f8a8af6b5dd0ab63f3f185d699900dd91a28a71"} Jul 21 00:55:56.830882 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:55:56.830828 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" podStartSLOduration=1.8308138330000001 podStartE2EDuration="1.830813833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:55:56.828334915 +0000 UTC m=+2351.043715239" watchObservedRunningTime="2026-07-21 00:55:56.830813833 +0000 UTC m=+2351.046194158" Jul 21 00:56:05.858343 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:05.858309 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:56:05.858808 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:05.858356 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:56:05.872022 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:05.871993 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:56:05.883046 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:05.883027 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:56:17.375644 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:17.375612 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:56:17.429847 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:17.429801 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:56:36.960134 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:36.960096 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:56:37.888705 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:37.888668 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4"] Jul 21 00:56:37.911152 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:37.911123 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:56:38.707831 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:38.707793 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:56:38.808639 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:38.808580 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:56:48.534882 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:48.534846 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:56:52.909615 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:52.909532 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" podUID="8cf717cf-7850-4039-a7c1-1ebb7d0f4b84" containerName="main" containerID="cri-o://aa554a11636b3b7dce6d614cb6edb4f5171719d10b51d166d8dacc4474a7a6f7" gracePeriod=45 Jul 21 00:56:53.132925 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:53.132888 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:56:53.139154 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:53.139120 2580 generic.go:356] "Generic (PLEG): container finished" podID="8cf717cf-7850-4039-a7c1-1ebb7d0f4b84" containerID="aa554a11636b3b7dce6d614cb6edb4f5171719d10b51d166d8dacc4474a7a6f7" exitCode=0 Jul 21 00:56:53.139323 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:53.139197 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" event={"ID":"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84","Type":"ContainerDied","Data":"aa554a11636b3b7dce6d614cb6edb4f5171719d10b51d166d8dacc4474a7a6f7"} Jul 21 00:56:53.165899 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:53.165793 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:56:53.215891 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:53.215855 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-dshm\") pod \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\" (UID: \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\") " Jul 21 00:56:53.216103 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:53.215910 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-tmp-dir\") pod \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\" (UID: \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\") " Jul 21 00:56:53.216103 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:53.215944 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-home\") pod \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\" (UID: \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\") " Jul 21 00:56:53.216103 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:53.215968 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-model-cache\") pod \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\" (UID: \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\") " Jul 21 00:56:53.216103 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:53.215999 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btwck\" (UniqueName: \"kubernetes.io/projected/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-kube-api-access-btwck\") pod \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\" (UID: \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\") " Jul 21 00:56:53.216337 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:53.216243 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-home" (OuterVolumeSpecName: "home") pod "8cf717cf-7850-4039-a7c1-1ebb7d0f4b84" (UID: "8cf717cf-7850-4039-a7c1-1ebb7d0f4b84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:56:53.216337 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:53.216284 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8cf717cf-7850-4039-a7c1-1ebb7d0f4b84" (UID: "8cf717cf-7850-4039-a7c1-1ebb7d0f4b84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:56:53.216337 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:53.216321 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-tls-certs\") pod \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\" (UID: \"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84\") " Jul 21 00:56:53.216513 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:53.216339 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-model-cache" (OuterVolumeSpecName: "model-cache") pod "8cf717cf-7850-4039-a7c1-1ebb7d0f4b84" (UID: "8cf717cf-7850-4039-a7c1-1ebb7d0f4b84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:56:53.216974 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:53.216774 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-tmp-dir\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:56:53.216974 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:53.216912 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-home\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:56:53.216974 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:53.216950 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-model-cache\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:56:53.218526 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:53.218498 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-dshm" (OuterVolumeSpecName: "dshm") pod "8cf717cf-7850-4039-a7c1-1ebb7d0f4b84" (UID: "8cf717cf-7850-4039-a7c1-1ebb7d0f4b84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:56:53.218654 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:53.218624 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-kube-api-access-btwck" (OuterVolumeSpecName: "kube-api-access-btwck") pod "8cf717cf-7850-4039-a7c1-1ebb7d0f4b84" (UID: "8cf717cf-7850-4039-a7c1-1ebb7d0f4b84"). InnerVolumeSpecName "kube-api-access-btwck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:56:53.218654 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:53.218634 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8cf717cf-7850-4039-a7c1-1ebb7d0f4b84" (UID: "8cf717cf-7850-4039-a7c1-1ebb7d0f4b84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 00:56:53.318317 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:53.318280 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-dshm\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:56:53.318317 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:53.318318 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btwck\" (UniqueName: \"kubernetes.io/projected/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-kube-api-access-btwck\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:56:53.318576 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:53.318332 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84-tls-certs\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:56:53.543786 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:53.543697 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:56:54.146265 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:54.146234 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" Jul 21 00:56:54.146737 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:54.146268 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4" event={"ID":"8cf717cf-7850-4039-a7c1-1ebb7d0f4b84","Type":"ContainerDied","Data":"e942eb6d6766bd9262b738722f8a8af6b5dd0ab63f3f185d699900dd91a28a71"} Jul 21 00:56:54.146737 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:54.146332 2580 scope.go:117] "RemoveContainer" containerID="aa554a11636b3b7dce6d614cb6edb4f5171719d10b51d166d8dacc4474a7a6f7" Jul 21 00:56:54.171674 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:54.171642 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4"] Jul 21 00:56:54.176587 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:54.176557 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ee9ef4/test-model-ee9ef4-kserve-7b6c75d6d9-499j4"] Jul 21 00:56:54.297713 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:54.297677 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cf717cf-7850-4039-a7c1-1ebb7d0f4b84" path="/var/lib/kubelet/pods/8cf717cf-7850-4039-a7c1-1ebb7d0f4b84/volumes" Jul 21 00:56:59.650660 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:59.650622 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:56:59.744936 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:59.744895 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:56:59.918016 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:59.917926 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7"] Jul 21 00:56:59.918173 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:59.918133 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" podUID="65509139-5b21-42bd-ba31-bb84b7fbb77f" containerName="istio-proxy" containerID="cri-o://721b56840b6e94d978fcd05dd0436c371856ffd4792b110adf43716933d8be0b" gracePeriod=30 Jul 21 00:56:59.954901 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:56:59.954865 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:00.209254 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:00.209164 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:00.888913 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:00.888870 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:00.990044 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:00.989998 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:05.165979 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.165950 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:57:05.210567 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.210465 2580 generic.go:356] "Generic (PLEG): container finished" podID="65509139-5b21-42bd-ba31-bb84b7fbb77f" containerID="721b56840b6e94d978fcd05dd0436c371856ffd4792b110adf43716933d8be0b" exitCode=0 Jul 21 00:57:05.210567 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.210525 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" Jul 21 00:57:05.210567 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.210547 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" event={"ID":"65509139-5b21-42bd-ba31-bb84b7fbb77f","Type":"ContainerDied","Data":"721b56840b6e94d978fcd05dd0436c371856ffd4792b110adf43716933d8be0b"} Jul 21 00:57:05.210810 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.210648 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7" event={"ID":"65509139-5b21-42bd-ba31-bb84b7fbb77f","Type":"ContainerDied","Data":"4536b61ed1b5bc59a966621a51b4ef0304a23bb94aefba3dc5ae159d60f8a86a"} Jul 21 00:57:05.210810 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.210674 2580 scope.go:117] "RemoveContainer" containerID="721b56840b6e94d978fcd05dd0436c371856ffd4792b110adf43716933d8be0b" Jul 21 00:57:05.220219 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.220199 2580 scope.go:117] "RemoveContainer" containerID="8786b9b57ba83073ba14b4f29bcf7a24cc85fff0bfb89b2df4675c5879ac4636" Jul 21 00:57:05.229204 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.229184 2580 scope.go:117] "RemoveContainer" containerID="721b56840b6e94d978fcd05dd0436c371856ffd4792b110adf43716933d8be0b" Jul 21 00:57:05.229514 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:57:05.229494 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"721b56840b6e94d978fcd05dd0436c371856ffd4792b110adf43716933d8be0b\": container with ID starting with 721b56840b6e94d978fcd05dd0436c371856ffd4792b110adf43716933d8be0b not found: ID does not exist" containerID="721b56840b6e94d978fcd05dd0436c371856ffd4792b110adf43716933d8be0b" Jul 21 00:57:05.229557 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.229527 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"721b56840b6e94d978fcd05dd0436c371856ffd4792b110adf43716933d8be0b"} err="failed to get container status \"721b56840b6e94d978fcd05dd0436c371856ffd4792b110adf43716933d8be0b\": rpc error: code = NotFound desc = could not find container \"721b56840b6e94d978fcd05dd0436c371856ffd4792b110adf43716933d8be0b\": container with ID starting with 721b56840b6e94d978fcd05dd0436c371856ffd4792b110adf43716933d8be0b not found: ID does not exist" Jul 21 00:57:05.229557 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.229548 2580 scope.go:117] "RemoveContainer" containerID="8786b9b57ba83073ba14b4f29bcf7a24cc85fff0bfb89b2df4675c5879ac4636" Jul 21 00:57:05.229857 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:57:05.229832 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8786b9b57ba83073ba14b4f29bcf7a24cc85fff0bfb89b2df4675c5879ac4636\": container with ID starting with 8786b9b57ba83073ba14b4f29bcf7a24cc85fff0bfb89b2df4675c5879ac4636 not found: ID does not exist" containerID="8786b9b57ba83073ba14b4f29bcf7a24cc85fff0bfb89b2df4675c5879ac4636" Jul 21 00:57:05.229897 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.229866 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8786b9b57ba83073ba14b4f29bcf7a24cc85fff0bfb89b2df4675c5879ac4636"} err="failed to get container status \"8786b9b57ba83073ba14b4f29bcf7a24cc85fff0bfb89b2df4675c5879ac4636\": rpc error: code = NotFound desc = could not find container \"8786b9b57ba83073ba14b4f29bcf7a24cc85fff0bfb89b2df4675c5879ac4636\": container with ID starting with 8786b9b57ba83073ba14b4f29bcf7a24cc85fff0bfb89b2df4675c5879ac4636 not found: ID does not exist" Jul 21 00:57:05.248269 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.248233 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-workload-socket\") pod \"65509139-5b21-42bd-ba31-bb84b7fbb77f\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " Jul 21 00:57:05.248269 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.248273 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-credential-socket\") pod \"65509139-5b21-42bd-ba31-bb84b7fbb77f\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " Jul 21 00:57:05.248475 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.248295 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65509139-5b21-42bd-ba31-bb84b7fbb77f-istiod-ca-cert\") pod \"65509139-5b21-42bd-ba31-bb84b7fbb77f\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " Jul 21 00:57:05.248475 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.248332 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65509139-5b21-42bd-ba31-bb84b7fbb77f-istio-podinfo\") pod \"65509139-5b21-42bd-ba31-bb84b7fbb77f\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " Jul 21 00:57:05.248475 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.248349 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kf6x\" (UniqueName: \"kubernetes.io/projected/65509139-5b21-42bd-ba31-bb84b7fbb77f-kube-api-access-5kf6x\") pod \"65509139-5b21-42bd-ba31-bb84b7fbb77f\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " Jul 21 00:57:05.248475 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.248402 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65509139-5b21-42bd-ba31-bb84b7fbb77f-istio-token\") pod \"65509139-5b21-42bd-ba31-bb84b7fbb77f\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " Jul 21 00:57:05.248475 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.248463 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-istio-data\") pod \"65509139-5b21-42bd-ba31-bb84b7fbb77f\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " Jul 21 00:57:05.248740 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.248491 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-istio-envoy\") pod \"65509139-5b21-42bd-ba31-bb84b7fbb77f\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " Jul 21 00:57:05.248740 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.248521 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-workload-certs\") pod \"65509139-5b21-42bd-ba31-bb84b7fbb77f\" (UID: \"65509139-5b21-42bd-ba31-bb84b7fbb77f\") " Jul 21 00:57:05.248740 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.248567 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "65509139-5b21-42bd-ba31-bb84b7fbb77f" (UID: "65509139-5b21-42bd-ba31-bb84b7fbb77f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:57:05.248740 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.248590 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "65509139-5b21-42bd-ba31-bb84b7fbb77f" (UID: "65509139-5b21-42bd-ba31-bb84b7fbb77f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:57:05.248947 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.248844 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-workload-socket\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:57:05.248947 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.248867 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-credential-socket\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:57:05.248947 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.248884 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65509139-5b21-42bd-ba31-bb84b7fbb77f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "65509139-5b21-42bd-ba31-bb84b7fbb77f" (UID: "65509139-5b21-42bd-ba31-bb84b7fbb77f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:57:05.249275 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.249178 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "65509139-5b21-42bd-ba31-bb84b7fbb77f" (UID: "65509139-5b21-42bd-ba31-bb84b7fbb77f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:57:05.249275 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.249320 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-istio-data" (OuterVolumeSpecName: "istio-data") pod "65509139-5b21-42bd-ba31-bb84b7fbb77f" (UID: "65509139-5b21-42bd-ba31-bb84b7fbb77f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:57:05.250941 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.250914 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "65509139-5b21-42bd-ba31-bb84b7fbb77f" (UID: "65509139-5b21-42bd-ba31-bb84b7fbb77f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:57:05.250941 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.250932 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65509139-5b21-42bd-ba31-bb84b7fbb77f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "65509139-5b21-42bd-ba31-bb84b7fbb77f" (UID: "65509139-5b21-42bd-ba31-bb84b7fbb77f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 00:57:05.251081 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.250939 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65509139-5b21-42bd-ba31-bb84b7fbb77f-istio-token" (OuterVolumeSpecName: "istio-token") pod "65509139-5b21-42bd-ba31-bb84b7fbb77f" (UID: "65509139-5b21-42bd-ba31-bb84b7fbb77f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:57:05.251177 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.251155 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65509139-5b21-42bd-ba31-bb84b7fbb77f-kube-api-access-5kf6x" (OuterVolumeSpecName: "kube-api-access-5kf6x") pod "65509139-5b21-42bd-ba31-bb84b7fbb77f" (UID: "65509139-5b21-42bd-ba31-bb84b7fbb77f"). InnerVolumeSpecName "kube-api-access-5kf6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:57:05.349713 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.349680 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65509139-5b21-42bd-ba31-bb84b7fbb77f-istiod-ca-cert\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:57:05.349713 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.349707 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65509139-5b21-42bd-ba31-bb84b7fbb77f-istio-podinfo\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:57:05.349713 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.349717 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kf6x\" (UniqueName: \"kubernetes.io/projected/65509139-5b21-42bd-ba31-bb84b7fbb77f-kube-api-access-5kf6x\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:57:05.349987 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.349728 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65509139-5b21-42bd-ba31-bb84b7fbb77f-istio-token\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:57:05.349987 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.349735 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-istio-data\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:57:05.349987 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.349743 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-istio-envoy\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:57:05.349987 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.349750 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65509139-5b21-42bd-ba31-bb84b7fbb77f-workload-certs\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:57:05.534030 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.533991 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7"] Jul 21 00:57:05.538168 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:05.538136 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1c3d09-openshift-default-b8658cbc7-578g7"] Jul 21 00:57:06.299052 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:06.299018 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65509139-5b21-42bd-ba31-bb84b7fbb77f" path="/var/lib/kubelet/pods/65509139-5b21-42bd-ba31-bb84b7fbb77f/volumes" Jul 21 00:57:07.894886 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:07.894772 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:10.789244 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:10.789195 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:15.414467 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:15.414435 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:15.822772 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:15.822725 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:22.242101 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:22.242066 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:22.340689 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:22.340651 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:22.538948 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:22.538863 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn"] Jul 21 00:57:22.539105 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:22.539069 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" podUID="59b77d75-567a-4112-b28a-15a134cd9bd5" containerName="istio-proxy" containerID="cri-o://7ec0c94720d2d482e22de44a98a23522a21a4ffb869562015d2f4b036314cb57" gracePeriod=30 Jul 21 00:57:22.590854 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:22.590812 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:22.776675 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:22.776592 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:23.177332 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.177294 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:23.513347 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.513256 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct"] Jul 21 00:57:23.513887 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.513750 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cf717cf-7850-4039-a7c1-1ebb7d0f4b84" containerName="main" Jul 21 00:57:23.513887 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.513764 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cf717cf-7850-4039-a7c1-1ebb7d0f4b84" containerName="main" Jul 21 00:57:23.513887 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.513783 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65509139-5b21-42bd-ba31-bb84b7fbb77f" containerName="istio-proxy" Jul 21 00:57:23.513887 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.513789 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="65509139-5b21-42bd-ba31-bb84b7fbb77f" containerName="istio-proxy" Jul 21 00:57:23.513887 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.513801 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65509139-5b21-42bd-ba31-bb84b7fbb77f" containerName="istio-proxy" Jul 21 00:57:23.513887 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.513807 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="65509139-5b21-42bd-ba31-bb84b7fbb77f" containerName="istio-proxy" Jul 21 00:57:23.513887 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.513887 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="65509139-5b21-42bd-ba31-bb84b7fbb77f" containerName="istio-proxy" Jul 21 00:57:23.514216 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.513897 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cf717cf-7850-4039-a7c1-1ebb7d0f4b84" containerName="main" Jul 21 00:57:23.517226 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.517209 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.521002 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.520977 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e541cf-openshift-default-dockercfg-rhcv7\"" Jul 21 00:57:23.521275 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.521253 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-credential-socket\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.521344 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.521295 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw97s\" (UniqueName: \"kubernetes.io/projected/615e5b88-e776-4b41-91fd-3fa9fe78a591-kube-api-access-sw97s\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.521344 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.521319 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-workload-socket\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.521430 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.521393 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-istio-envoy\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.521482 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.521450 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-istio-data\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.521522 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.521499 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615e5b88-e776-4b41-91fd-3fa9fe78a591-istio-podinfo\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.521573 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.521528 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615e5b88-e776-4b41-91fd-3fa9fe78a591-istiod-ca-cert\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.521573 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.521561 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/615e5b88-e776-4b41-91fd-3fa9fe78a591-istio-token\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.521718 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.521692 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-workload-certs\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.536655 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.536624 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct"] Jul 21 00:57:23.540241 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.540199 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:23.623234 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.623199 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615e5b88-e776-4b41-91fd-3fa9fe78a591-istiod-ca-cert\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.623437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.623246 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/615e5b88-e776-4b41-91fd-3fa9fe78a591-istio-token\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.623437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.623348 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-workload-certs\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.623437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.623391 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-credential-socket\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.623437 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.623421 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw97s\" (UniqueName: \"kubernetes.io/projected/615e5b88-e776-4b41-91fd-3fa9fe78a591-kube-api-access-sw97s\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.623711 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.623448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-workload-socket\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.623711 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.623572 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-istio-envoy\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.623711 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.623646 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-istio-data\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.623711 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.623697 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615e5b88-e776-4b41-91fd-3fa9fe78a591-istio-podinfo\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.623915 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.623762 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-workload-certs\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.623915 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.623842 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-credential-socket\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.624100 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.624074 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615e5b88-e776-4b41-91fd-3fa9fe78a591-istiod-ca-cert\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.624157 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.624087 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-istio-data\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.624198 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.624153 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-workload-socket\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.626028 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.626007 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-istio-envoy\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.626245 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.626226 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615e5b88-e776-4b41-91fd-3fa9fe78a591-istio-podinfo\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.632291 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.632260 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/615e5b88-e776-4b41-91fd-3fa9fe78a591-istio-token\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.632464 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.632446 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw97s\" (UniqueName: \"kubernetes.io/projected/615e5b88-e776-4b41-91fd-3fa9fe78a591-kube-api-access-sw97s\") pod \"e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.827629 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.827567 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:23.984778 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.984744 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct"] Jul 21 00:57:23.985933 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:57:23.985906 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615e5b88_e776_4b41_91fd_3fa9fe78a591.slice/crio-1f20fcb56064dfb2601e870abecf7f95fdbc00219ba0452e05a3bd2b98aa2c7c WatchSource:0}: Error finding container 1f20fcb56064dfb2601e870abecf7f95fdbc00219ba0452e05a3bd2b98aa2c7c: Status 404 returned error can't find the container with id 1f20fcb56064dfb2601e870abecf7f95fdbc00219ba0452e05a3bd2b98aa2c7c Jul 21 00:57:23.988613 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.988550 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:57:23.988731 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.988662 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:57:23.988731 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:23.988708 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:57:24.075329 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:24.075291 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:24.321615 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:24.321563 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" event={"ID":"615e5b88-e776-4b41-91fd-3fa9fe78a591","Type":"ContainerStarted","Data":"90929ce233b393d65ced1fc63eee3ecde31b6f78c1f59731d1beed55450ac924"} Jul 21 00:57:24.321798 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:24.321648 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" event={"ID":"615e5b88-e776-4b41-91fd-3fa9fe78a591","Type":"ContainerStarted","Data":"1f20fcb56064dfb2601e870abecf7f95fdbc00219ba0452e05a3bd2b98aa2c7c"} Jul 21 00:57:24.345079 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:24.345025 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" podStartSLOduration=1.345007181 podStartE2EDuration="1.345007181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:57:24.343084526 +0000 UTC m=+2438.558464851" watchObservedRunningTime="2026-07-21 00:57:24.345007181 +0000 UTC m=+2438.560387564" Jul 21 00:57:24.828166 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:24.828129 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:25.833578 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:25.833546 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:26.335672 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:26.335638 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:26.337070 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:26.337044 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:26.347814 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:26.347786 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:26.443946 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:26.443902 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:26.537127 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:26.537095 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:26.640966 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:26.640875 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:27.250012 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.249973 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:27.349197 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.349166 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:27.799064 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.799034 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:57:27.865867 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.865834 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-credential-socket\") pod \"59b77d75-567a-4112-b28a-15a134cd9bd5\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " Jul 21 00:57:27.866052 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.865892 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b77d75-567a-4112-b28a-15a134cd9bd5-istio-podinfo\") pod \"59b77d75-567a-4112-b28a-15a134cd9bd5\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " Jul 21 00:57:27.866052 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.865917 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-istio-data\") pod \"59b77d75-567a-4112-b28a-15a134cd9bd5\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " Jul 21 00:57:27.866052 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.865982 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b77d75-567a-4112-b28a-15a134cd9bd5-istio-token\") pod \"59b77d75-567a-4112-b28a-15a134cd9bd5\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " Jul 21 00:57:27.866052 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.866025 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-istio-envoy\") pod \"59b77d75-567a-4112-b28a-15a134cd9bd5\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " Jul 21 00:57:27.866239 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.866059 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b77d75-567a-4112-b28a-15a134cd9bd5-istiod-ca-cert\") pod \"59b77d75-567a-4112-b28a-15a134cd9bd5\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " Jul 21 00:57:27.866239 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.866085 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlzcg\" (UniqueName: \"kubernetes.io/projected/59b77d75-567a-4112-b28a-15a134cd9bd5-kube-api-access-qlzcg\") pod \"59b77d75-567a-4112-b28a-15a134cd9bd5\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " Jul 21 00:57:27.866239 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.866110 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-workload-socket\") pod \"59b77d75-567a-4112-b28a-15a134cd9bd5\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " Jul 21 00:57:27.866239 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.866151 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-workload-certs\") pod \"59b77d75-567a-4112-b28a-15a134cd9bd5\" (UID: \"59b77d75-567a-4112-b28a-15a134cd9bd5\") " Jul 21 00:57:27.866239 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.866153 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "59b77d75-567a-4112-b28a-15a134cd9bd5" (UID: "59b77d75-567a-4112-b28a-15a134cd9bd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:57:27.866490 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.866377 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "59b77d75-567a-4112-b28a-15a134cd9bd5" (UID: "59b77d75-567a-4112-b28a-15a134cd9bd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:57:27.866490 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.866469 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59b77d75-567a-4112-b28a-15a134cd9bd5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "59b77d75-567a-4112-b28a-15a134cd9bd5" (UID: "59b77d75-567a-4112-b28a-15a134cd9bd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:57:27.866594 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.866474 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-workload-socket\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:57:27.866739 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.866623 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "59b77d75-567a-4112-b28a-15a134cd9bd5" (UID: "59b77d75-567a-4112-b28a-15a134cd9bd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:57:27.866739 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.866657 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-credential-socket\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:57:27.867182 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.867151 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-istio-data" (OuterVolumeSpecName: "istio-data") pod "59b77d75-567a-4112-b28a-15a134cd9bd5" (UID: "59b77d75-567a-4112-b28a-15a134cd9bd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:57:27.868331 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.868308 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59b77d75-567a-4112-b28a-15a134cd9bd5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "59b77d75-567a-4112-b28a-15a134cd9bd5" (UID: "59b77d75-567a-4112-b28a-15a134cd9bd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 00:57:27.868514 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.868493 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b77d75-567a-4112-b28a-15a134cd9bd5-kube-api-access-qlzcg" (OuterVolumeSpecName: "kube-api-access-qlzcg") pod "59b77d75-567a-4112-b28a-15a134cd9bd5" (UID: "59b77d75-567a-4112-b28a-15a134cd9bd5"). InnerVolumeSpecName "kube-api-access-qlzcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:57:27.868928 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.868897 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b77d75-567a-4112-b28a-15a134cd9bd5-istio-token" (OuterVolumeSpecName: "istio-token") pod "59b77d75-567a-4112-b28a-15a134cd9bd5" (UID: "59b77d75-567a-4112-b28a-15a134cd9bd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:57:27.869023 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.868937 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "59b77d75-567a-4112-b28a-15a134cd9bd5" (UID: "59b77d75-567a-4112-b28a-15a134cd9bd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:57:27.967725 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.967682 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-workload-certs\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:57:27.967725 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.967717 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59b77d75-567a-4112-b28a-15a134cd9bd5-istio-podinfo\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:57:27.967725 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.967726 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-istio-data\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:57:27.967725 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.967734 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59b77d75-567a-4112-b28a-15a134cd9bd5-istio-token\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:57:27.968019 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.967741 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59b77d75-567a-4112-b28a-15a134cd9bd5-istio-envoy\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:57:27.968019 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.967748 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59b77d75-567a-4112-b28a-15a134cd9bd5-istiod-ca-cert\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:57:27.968019 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:27.967757 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlzcg\" (UniqueName: \"kubernetes.io/projected/59b77d75-567a-4112-b28a-15a134cd9bd5-kube-api-access-qlzcg\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:57:28.348611 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:28.348561 2580 generic.go:356] "Generic (PLEG): container finished" podID="59b77d75-567a-4112-b28a-15a134cd9bd5" containerID="7ec0c94720d2d482e22de44a98a23522a21a4ffb869562015d2f4b036314cb57" exitCode=0 Jul 21 00:57:28.349014 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:28.348645 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" Jul 21 00:57:28.349014 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:28.348655 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" event={"ID":"59b77d75-567a-4112-b28a-15a134cd9bd5","Type":"ContainerDied","Data":"7ec0c94720d2d482e22de44a98a23522a21a4ffb869562015d2f4b036314cb57"} Jul 21 00:57:28.349014 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:28.348722 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn" event={"ID":"59b77d75-567a-4112-b28a-15a134cd9bd5","Type":"ContainerDied","Data":"d2820f80d61d36e31249c010d9dc5bcfc5d0b8671e1987db2e0b4c93a9206f03"} Jul 21 00:57:28.349014 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:28.348755 2580 scope.go:117] "RemoveContainer" containerID="7ec0c94720d2d482e22de44a98a23522a21a4ffb869562015d2f4b036314cb57" Jul 21 00:57:28.358442 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:28.358420 2580 scope.go:117] "RemoveContainer" containerID="92a0eda9a8497d6a0b3e7e1e0f4ef7c8ab27063e686feea17730e0a2e0edb044" Jul 21 00:57:28.367362 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:28.367343 2580 scope.go:117] "RemoveContainer" containerID="7ec0c94720d2d482e22de44a98a23522a21a4ffb869562015d2f4b036314cb57" Jul 21 00:57:28.367685 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:57:28.367666 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ec0c94720d2d482e22de44a98a23522a21a4ffb869562015d2f4b036314cb57\": container with ID starting with 7ec0c94720d2d482e22de44a98a23522a21a4ffb869562015d2f4b036314cb57 not found: ID does not exist" containerID="7ec0c94720d2d482e22de44a98a23522a21a4ffb869562015d2f4b036314cb57" Jul 21 00:57:28.367742 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:28.367695 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ec0c94720d2d482e22de44a98a23522a21a4ffb869562015d2f4b036314cb57"} err="failed to get container status \"7ec0c94720d2d482e22de44a98a23522a21a4ffb869562015d2f4b036314cb57\": rpc error: code = NotFound desc = could not find container \"7ec0c94720d2d482e22de44a98a23522a21a4ffb869562015d2f4b036314cb57\": container with ID starting with 7ec0c94720d2d482e22de44a98a23522a21a4ffb869562015d2f4b036314cb57 not found: ID does not exist" Jul 21 00:57:28.367742 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:28.367715 2580 scope.go:117] "RemoveContainer" containerID="92a0eda9a8497d6a0b3e7e1e0f4ef7c8ab27063e686feea17730e0a2e0edb044" Jul 21 00:57:28.367986 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:57:28.367967 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92a0eda9a8497d6a0b3e7e1e0f4ef7c8ab27063e686feea17730e0a2e0edb044\": container with ID starting with 92a0eda9a8497d6a0b3e7e1e0f4ef7c8ab27063e686feea17730e0a2e0edb044 not found: ID does not exist" containerID="92a0eda9a8497d6a0b3e7e1e0f4ef7c8ab27063e686feea17730e0a2e0edb044" Jul 21 00:57:28.368053 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:28.367993 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a0eda9a8497d6a0b3e7e1e0f4ef7c8ab27063e686feea17730e0a2e0edb044"} err="failed to get container status \"92a0eda9a8497d6a0b3e7e1e0f4ef7c8ab27063e686feea17730e0a2e0edb044\": rpc error: code = NotFound desc = could not find container \"92a0eda9a8497d6a0b3e7e1e0f4ef7c8ab27063e686feea17730e0a2e0edb044\": container with ID starting with 92a0eda9a8497d6a0b3e7e1e0f4ef7c8ab27063e686feea17730e0a2e0edb044 not found: ID does not exist" Jul 21 00:57:28.381710 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:28.381675 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn"] Jul 21 00:57:28.393204 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:28.393173 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ee9ef4-openshift-default-7fbfd88cdc-h64qn"] Jul 21 00:57:30.298029 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:30.297997 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59b77d75-567a-4112-b28a-15a134cd9bd5" path="/var/lib/kubelet/pods/59b77d75-567a-4112-b28a-15a134cd9bd5/volumes" Jul 21 00:57:30.858537 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:30.858509 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:31.365799 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.365766 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:31.653027 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.652943 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns"] Jul 21 00:57:31.653566 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.653547 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59b77d75-567a-4112-b28a-15a134cd9bd5" containerName="istio-proxy" Jul 21 00:57:31.653634 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.653571 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="59b77d75-567a-4112-b28a-15a134cd9bd5" containerName="istio-proxy" Jul 21 00:57:31.653737 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.653725 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="65509139-5b21-42bd-ba31-bb84b7fbb77f" containerName="istio-proxy" Jul 21 00:57:31.653769 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.653740 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="59b77d75-567a-4112-b28a-15a134cd9bd5" containerName="istio-proxy" Jul 21 00:57:31.653769 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.653747 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="59b77d75-567a-4112-b28a-15a134cd9bd5" containerName="istio-proxy" Jul 21 00:57:31.658429 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.658401 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.661757 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.661736 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a3348a-openshift-default-dockercfg-9hn49\"" Jul 21 00:57:31.685728 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.685696 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns"] Jul 21 00:57:31.697377 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.697341 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:31.703142 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.703113 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-credential-socket\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.703316 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.703163 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-istio-data\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.703316 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.703198 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-istio-envoy\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.703316 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.703233 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-workload-certs\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.703316 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.703295 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9823a172-2287-484e-a10f-f2c6e27d0e19-istio-token\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.703522 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.703328 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-workload-socket\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.703522 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.703409 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9823a172-2287-484e-a10f-f2c6e27d0e19-istiod-ca-cert\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.703522 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.703462 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9823a172-2287-484e-a10f-f2c6e27d0e19-istio-podinfo\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.703522 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.703496 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpbml\" (UniqueName: \"kubernetes.io/projected/9823a172-2287-484e-a10f-f2c6e27d0e19-kube-api-access-dpbml\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.787194 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.787153 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:31.804064 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.804007 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9823a172-2287-484e-a10f-f2c6e27d0e19-istio-token\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.804064 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.804068 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-workload-socket\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.804311 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.804104 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9823a172-2287-484e-a10f-f2c6e27d0e19-istiod-ca-cert\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.804311 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.804136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9823a172-2287-484e-a10f-f2c6e27d0e19-istio-podinfo\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.804311 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.804163 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpbml\" (UniqueName: \"kubernetes.io/projected/9823a172-2287-484e-a10f-f2c6e27d0e19-kube-api-access-dpbml\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.804311 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.804214 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-credential-socket\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.804311 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.804268 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-istio-data\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.804311 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.804306 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-istio-envoy\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.804727 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.804347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-workload-certs\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.804727 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.804578 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-workload-socket\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.804727 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.804672 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-istio-data\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.804874 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.804724 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-credential-socket\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.804874 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.804796 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-workload-certs\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.805062 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.805044 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9823a172-2287-484e-a10f-f2c6e27d0e19-istiod-ca-cert\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.806870 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.806853 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9823a172-2287-484e-a10f-f2c6e27d0e19-istio-podinfo\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.806961 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.806785 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-istio-envoy\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.818114 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.818082 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9823a172-2287-484e-a10f-f2c6e27d0e19-istio-token\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.818269 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.818149 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpbml\" (UniqueName: \"kubernetes.io/projected/9823a172-2287-484e-a10f-f2c6e27d0e19-kube-api-access-dpbml\") pod \"e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:31.968478 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:31.968372 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:32.125660 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:32.125624 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns"] Jul 21 00:57:32.126770 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:57:32.126742 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9823a172_2287_484e_a10f_f2c6e27d0e19.slice/crio-ab446cfa8bb56e3a31be6e25dc0db9c572cab4c111ae03a2aad10ec6279c34f4 WatchSource:0}: Error finding container ab446cfa8bb56e3a31be6e25dc0db9c572cab4c111ae03a2aad10ec6279c34f4: Status 404 returned error can't find the container with id ab446cfa8bb56e3a31be6e25dc0db9c572cab4c111ae03a2aad10ec6279c34f4 Jul 21 00:57:32.129151 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:32.129115 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:57:32.129251 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:32.129205 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:57:32.129321 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:32.129248 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:57:32.376793 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:32.376753 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" event={"ID":"9823a172-2287-484e-a10f-f2c6e27d0e19","Type":"ContainerStarted","Data":"514d8bb4ef3a582c64b9e54a56e6cf13ac5b11f674d03f51dbf8e81c9b51c558"} Jul 21 00:57:32.377184 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:32.376816 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" event={"ID":"9823a172-2287-484e-a10f-f2c6e27d0e19","Type":"ContainerStarted","Data":"ab446cfa8bb56e3a31be6e25dc0db9c572cab4c111ae03a2aad10ec6279c34f4"} Jul 21 00:57:32.399294 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:32.399234 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" podStartSLOduration=1.399213313 podStartE2EDuration="1.399213313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:57:32.39648147 +0000 UTC m=+2446.611861826" watchObservedRunningTime="2026-07-21 00:57:32.399213313 +0000 UTC m=+2446.614593637" Jul 21 00:57:32.969372 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:32.969332 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:33.974044 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:33.974010 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:34.384717 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:34.384684 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:34.401116 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:34.399860 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:34.403950 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:34.403916 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:34.505157 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:34.505059 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:34.761011 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:34.760885 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:34.851221 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:34.851177 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:35.346207 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:35.346150 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:35.445670 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:35.445628 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:38.906146 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:38.906092 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:41.337097 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:41.337061 2580 patch_prober.go:28] interesting pod/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Jul 21 00:57:41.337534 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:41.337123 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" podUID="615e5b88-e776-4b41-91fd-3fa9fe78a591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Jul 21 00:57:41.452858 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:41.452822 2580 generic.go:356] "Generic (PLEG): container finished" podID="615e5b88-e776-4b41-91fd-3fa9fe78a591" containerID="90929ce233b393d65ced1fc63eee3ecde31b6f78c1f59731d1beed55450ac924" exitCode=0 Jul 21 00:57:41.453040 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:41.452889 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" event={"ID":"615e5b88-e776-4b41-91fd-3fa9fe78a591","Type":"ContainerDied","Data":"90929ce233b393d65ced1fc63eee3ecde31b6f78c1f59731d1beed55450ac924"} Jul 21 00:57:41.453303 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:41.453289 2580 scope.go:117] "RemoveContainer" containerID="90929ce233b393d65ced1fc63eee3ecde31b6f78c1f59731d1beed55450ac924" Jul 21 00:57:41.454419 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:41.454390 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:57:41.454519 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:41.454459 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:57:41.454519 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:41.454489 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:57:42.465747 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:42.465704 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" event={"ID":"615e5b88-e776-4b41-91fd-3fa9fe78a591","Type":"ContainerStarted","Data":"f7ae3d212085d760343ef59599e7b96e5ac186fbd58eaec023e9fb25157528dd"} Jul 21 00:57:42.828454 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:42.828405 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:42.828688 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:42.828665 2580 patch_prober.go:28] interesting pod/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Jul 21 00:57:42.828740 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:42.828715 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" podUID="615e5b88-e776-4b41-91fd-3fa9fe78a591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Jul 21 00:57:43.828920 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:43.828880 2580 patch_prober.go:28] interesting pod/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Jul 21 00:57:43.829328 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:43.828949 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" podUID="615e5b88-e776-4b41-91fd-3fa9fe78a591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Jul 21 00:57:44.828139 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:44.828096 2580 patch_prober.go:28] interesting pod/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Jul 21 00:57:44.828324 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:44.828177 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" podUID="615e5b88-e776-4b41-91fd-3fa9fe78a591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Jul 21 00:57:45.827990 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:45.827949 2580 patch_prober.go:28] interesting pod/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Jul 21 00:57:45.828378 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:45.828038 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" podUID="615e5b88-e776-4b41-91fd-3fa9fe78a591" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Jul 21 00:57:46.496203 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:46.496104 2580 generic.go:356] "Generic (PLEG): container finished" podID="9823a172-2287-484e-a10f-f2c6e27d0e19" containerID="514d8bb4ef3a582c64b9e54a56e6cf13ac5b11f674d03f51dbf8e81c9b51c558" exitCode=0 Jul 21 00:57:46.496203 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:46.496166 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" event={"ID":"9823a172-2287-484e-a10f-f2c6e27d0e19","Type":"ContainerDied","Data":"514d8bb4ef3a582c64b9e54a56e6cf13ac5b11f674d03f51dbf8e81c9b51c558"} Jul 21 00:57:46.496710 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:46.496693 2580 scope.go:117] "RemoveContainer" containerID="514d8bb4ef3a582c64b9e54a56e6cf13ac5b11f674d03f51dbf8e81c9b51c558" Jul 21 00:57:46.497937 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:46.497899 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:57:46.498034 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:46.498015 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:57:46.498105 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:46.498090 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:57:46.832289 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:46.832260 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:46.832793 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:46.832737 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:46.833612 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:46.833575 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:57:47.510211 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:47.510164 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" event={"ID":"9823a172-2287-484e-a10f-f2c6e27d0e19","Type":"ContainerStarted","Data":"4fb31371dc260cd32ceee0a71b88080bce4488193add1e85a8e848b26f2d5fad"} Jul 21 00:57:47.719680 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:47.719639 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:47.823122 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:47.823084 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:47.968938 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:47.968911 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:47.970206 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:47.970179 2580 patch_prober.go:28] interesting pod/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 21 00:57:47.970321 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:47.970239 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" podUID="9823a172-2287-484e-a10f-f2c6e27d0e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 21 00:57:48.969386 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:48.969340 2580 patch_prober.go:28] interesting pod/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 21 00:57:48.969868 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:48.969416 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" podUID="9823a172-2287-484e-a10f-f2c6e27d0e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 21 00:57:49.400731 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:49.400679 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:49.969661 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:49.969585 2580 patch_prober.go:28] interesting pod/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 21 00:57:49.970071 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:49.969682 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" podUID="9823a172-2287-484e-a10f-f2c6e27d0e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 21 00:57:50.969486 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:50.969445 2580 patch_prober.go:28] interesting pod/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Jul 21 00:57:50.969692 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:50.969519 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" podUID="9823a172-2287-484e-a10f-f2c6e27d0e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Jul 21 00:57:51.972842 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:51.972811 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:51.973740 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:51.973719 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:57:57.558351 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:57.558314 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:57:57.661351 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:57:57.661303 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:02.850326 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:02.850289 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:07.189340 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:07.189298 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:07.593552 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:07.593502 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:10.844470 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:10.844427 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:11.197391 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:11.197313 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:13.634494 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:13.634447 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:13.735337 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:13.735299 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:13.878266 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:13.878231 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct"] Jul 21 00:58:13.878488 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:13.878460 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" podUID="615e5b88-e776-4b41-91fd-3fa9fe78a591" containerName="istio-proxy" containerID="cri-o://f7ae3d212085d760343ef59599e7b96e5ac186fbd58eaec023e9fb25157528dd" gracePeriod=30 Jul 21 00:58:13.895787 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:13.895704 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:14.149515 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:14.149425 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:16.835024 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:16.834984 2580 patch_prober.go:28] interesting pod/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 00:58:16.835439 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:16.835060 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" podUID="615e5b88-e776-4b41-91fd-3fa9fe78a591" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 00:58:19.141517 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.141485 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:58:19.187572 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.187485 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-workload-certs\") pod \"615e5b88-e776-4b41-91fd-3fa9fe78a591\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " Jul 21 00:58:19.187572 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.187542 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615e5b88-e776-4b41-91fd-3fa9fe78a591-istiod-ca-cert\") pod \"615e5b88-e776-4b41-91fd-3fa9fe78a591\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " Jul 21 00:58:19.187811 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.187581 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sw97s\" (UniqueName: \"kubernetes.io/projected/615e5b88-e776-4b41-91fd-3fa9fe78a591-kube-api-access-sw97s\") pod \"615e5b88-e776-4b41-91fd-3fa9fe78a591\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " Jul 21 00:58:19.187811 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.187643 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/615e5b88-e776-4b41-91fd-3fa9fe78a591-istio-token\") pod \"615e5b88-e776-4b41-91fd-3fa9fe78a591\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " Jul 21 00:58:19.187811 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.187699 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-istio-data\") pod \"615e5b88-e776-4b41-91fd-3fa9fe78a591\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " Jul 21 00:58:19.187811 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.187741 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615e5b88-e776-4b41-91fd-3fa9fe78a591-istio-podinfo\") pod \"615e5b88-e776-4b41-91fd-3fa9fe78a591\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " Jul 21 00:58:19.187811 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.187766 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-istio-envoy\") pod \"615e5b88-e776-4b41-91fd-3fa9fe78a591\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " Jul 21 00:58:19.187811 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.187801 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-credential-socket\") pod \"615e5b88-e776-4b41-91fd-3fa9fe78a591\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " Jul 21 00:58:19.188111 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.187828 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-workload-socket\") pod \"615e5b88-e776-4b41-91fd-3fa9fe78a591\" (UID: \"615e5b88-e776-4b41-91fd-3fa9fe78a591\") " Jul 21 00:58:19.188111 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.187831 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "615e5b88-e776-4b41-91fd-3fa9fe78a591" (UID: "615e5b88-e776-4b41-91fd-3fa9fe78a591"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:58:19.188111 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.187993 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/615e5b88-e776-4b41-91fd-3fa9fe78a591-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "615e5b88-e776-4b41-91fd-3fa9fe78a591" (UID: "615e5b88-e776-4b41-91fd-3fa9fe78a591"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:58:19.188275 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.188156 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "615e5b88-e776-4b41-91fd-3fa9fe78a591" (UID: "615e5b88-e776-4b41-91fd-3fa9fe78a591"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:58:19.188404 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.188365 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "615e5b88-e776-4b41-91fd-3fa9fe78a591" (UID: "615e5b88-e776-4b41-91fd-3fa9fe78a591"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:58:19.188879 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.188857 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-workload-certs\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:58:19.189060 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.189024 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/615e5b88-e776-4b41-91fd-3fa9fe78a591-istiod-ca-cert\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:58:19.189060 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.189047 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-credential-socket\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:58:19.189229 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.189063 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-workload-socket\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:58:19.189229 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.188963 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-istio-data" (OuterVolumeSpecName: "istio-data") pod "615e5b88-e776-4b41-91fd-3fa9fe78a591" (UID: "615e5b88-e776-4b41-91fd-3fa9fe78a591"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:58:19.190363 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.190333 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615e5b88-e776-4b41-91fd-3fa9fe78a591-kube-api-access-sw97s" (OuterVolumeSpecName: "kube-api-access-sw97s") pod "615e5b88-e776-4b41-91fd-3fa9fe78a591" (UID: "615e5b88-e776-4b41-91fd-3fa9fe78a591"). InnerVolumeSpecName "kube-api-access-sw97s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:58:19.190443 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.190351 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "615e5b88-e776-4b41-91fd-3fa9fe78a591" (UID: "615e5b88-e776-4b41-91fd-3fa9fe78a591"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:58:19.190783 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.190755 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615e5b88-e776-4b41-91fd-3fa9fe78a591-istio-token" (OuterVolumeSpecName: "istio-token") pod "615e5b88-e776-4b41-91fd-3fa9fe78a591" (UID: "615e5b88-e776-4b41-91fd-3fa9fe78a591"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:58:19.190783 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.190755 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/615e5b88-e776-4b41-91fd-3fa9fe78a591-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "615e5b88-e776-4b41-91fd-3fa9fe78a591" (UID: "615e5b88-e776-4b41-91fd-3fa9fe78a591"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 00:58:19.289875 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.289836 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw97s\" (UniqueName: \"kubernetes.io/projected/615e5b88-e776-4b41-91fd-3fa9fe78a591-kube-api-access-sw97s\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:58:19.289875 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.289871 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/615e5b88-e776-4b41-91fd-3fa9fe78a591-istio-token\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:58:19.289875 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.289881 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-istio-data\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:58:19.289875 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.289889 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/615e5b88-e776-4b41-91fd-3fa9fe78a591-istio-podinfo\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:58:19.290157 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.289897 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/615e5b88-e776-4b41-91fd-3fa9fe78a591-istio-envoy\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:58:19.700922 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.700885 2580 generic.go:356] "Generic (PLEG): container finished" podID="615e5b88-e776-4b41-91fd-3fa9fe78a591" containerID="f7ae3d212085d760343ef59599e7b96e5ac186fbd58eaec023e9fb25157528dd" exitCode=0 Jul 21 00:58:19.701146 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.700946 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" Jul 21 00:58:19.701146 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.700975 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" event={"ID":"615e5b88-e776-4b41-91fd-3fa9fe78a591","Type":"ContainerDied","Data":"f7ae3d212085d760343ef59599e7b96e5ac186fbd58eaec023e9fb25157528dd"} Jul 21 00:58:19.701146 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.701046 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct" event={"ID":"615e5b88-e776-4b41-91fd-3fa9fe78a591","Type":"ContainerDied","Data":"1f20fcb56064dfb2601e870abecf7f95fdbc00219ba0452e05a3bd2b98aa2c7c"} Jul 21 00:58:19.701146 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.701078 2580 scope.go:117] "RemoveContainer" containerID="f7ae3d212085d760343ef59599e7b96e5ac186fbd58eaec023e9fb25157528dd" Jul 21 00:58:19.711484 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.711464 2580 scope.go:117] "RemoveContainer" containerID="90929ce233b393d65ced1fc63eee3ecde31b6f78c1f59731d1beed55450ac924" Jul 21 00:58:19.721448 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.721424 2580 scope.go:117] "RemoveContainer" containerID="f7ae3d212085d760343ef59599e7b96e5ac186fbd58eaec023e9fb25157528dd" Jul 21 00:58:19.721815 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:58:19.721790 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7ae3d212085d760343ef59599e7b96e5ac186fbd58eaec023e9fb25157528dd\": container with ID starting with f7ae3d212085d760343ef59599e7b96e5ac186fbd58eaec023e9fb25157528dd not found: ID does not exist" containerID="f7ae3d212085d760343ef59599e7b96e5ac186fbd58eaec023e9fb25157528dd" Jul 21 00:58:19.721918 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.721823 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7ae3d212085d760343ef59599e7b96e5ac186fbd58eaec023e9fb25157528dd"} err="failed to get container status \"f7ae3d212085d760343ef59599e7b96e5ac186fbd58eaec023e9fb25157528dd\": rpc error: code = NotFound desc = could not find container \"f7ae3d212085d760343ef59599e7b96e5ac186fbd58eaec023e9fb25157528dd\": container with ID starting with f7ae3d212085d760343ef59599e7b96e5ac186fbd58eaec023e9fb25157528dd not found: ID does not exist" Jul 21 00:58:19.721918 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.721843 2580 scope.go:117] "RemoveContainer" containerID="90929ce233b393d65ced1fc63eee3ecde31b6f78c1f59731d1beed55450ac924" Jul 21 00:58:19.722153 ip-10-0-139-117 kubenswrapper[2580]: E0721 00:58:19.722122 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90929ce233b393d65ced1fc63eee3ecde31b6f78c1f59731d1beed55450ac924\": container with ID starting with 90929ce233b393d65ced1fc63eee3ecde31b6f78c1f59731d1beed55450ac924 not found: ID does not exist" containerID="90929ce233b393d65ced1fc63eee3ecde31b6f78c1f59731d1beed55450ac924" Jul 21 00:58:19.722195 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.722162 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90929ce233b393d65ced1fc63eee3ecde31b6f78c1f59731d1beed55450ac924"} err="failed to get container status \"90929ce233b393d65ced1fc63eee3ecde31b6f78c1f59731d1beed55450ac924\": rpc error: code = NotFound desc = could not find container \"90929ce233b393d65ced1fc63eee3ecde31b6f78c1f59731d1beed55450ac924\": container with ID starting with 90929ce233b393d65ced1fc63eee3ecde31b6f78c1f59731d1beed55450ac924 not found: ID does not exist" Jul 21 00:58:19.725441 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.725386 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct"] Jul 21 00:58:19.731807 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:19.731711 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e541cf-openshift-default-5494d4b5cf-7sfct"] Jul 21 00:58:20.297392 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:20.297353 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="615e5b88-e776-4b41-91fd-3fa9fe78a591" path="/var/lib/kubelet/pods/615e5b88-e776-4b41-91fd-3fa9fe78a591/volumes" Jul 21 00:58:22.728712 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:22.728677 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:29.735614 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:29.735561 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:34.349997 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:34.349949 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:34.762129 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:34.762019 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:40.446960 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:40.446859 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:40.546827 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:40.546793 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:40.700999 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:40.700908 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns"] Jul 21 00:58:40.701230 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:40.701174 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" podUID="9823a172-2287-484e-a10f-f2c6e27d0e19" containerName="istio-proxy" containerID="cri-o://4fb31371dc260cd32ceee0a71b88080bce4488193add1e85a8e848b26f2d5fad" gracePeriod=30 Jul 21 00:58:40.724670 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:40.724640 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:41.005167 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.005056 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:41.343611 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.343547 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:41.651527 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.651443 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9"] Jul 21 00:58:41.651986 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.651968 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="615e5b88-e776-4b41-91fd-3fa9fe78a591" containerName="istio-proxy" Jul 21 00:58:41.652034 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.651990 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="615e5b88-e776-4b41-91fd-3fa9fe78a591" containerName="istio-proxy" Jul 21 00:58:41.652034 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.652004 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59b77d75-567a-4112-b28a-15a134cd9bd5" containerName="istio-proxy" Jul 21 00:58:41.652034 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.652013 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="59b77d75-567a-4112-b28a-15a134cd9bd5" containerName="istio-proxy" Jul 21 00:58:41.652172 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.652034 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="615e5b88-e776-4b41-91fd-3fa9fe78a591" containerName="istio-proxy" Jul 21 00:58:41.652172 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.652045 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="615e5b88-e776-4b41-91fd-3fa9fe78a591" containerName="istio-proxy" Jul 21 00:58:41.652363 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.652331 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="615e5b88-e776-4b41-91fd-3fa9fe78a591" containerName="istio-proxy" Jul 21 00:58:41.652546 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.652526 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="615e5b88-e776-4b41-91fd-3fa9fe78a591" containerName="istio-proxy" Jul 21 00:58:41.655679 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.655662 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.658279 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.658261 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-16b734-openshift-default-dockercfg-r5x6h\"" Jul 21 00:58:41.669508 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.669468 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9"] Jul 21 00:58:41.684371 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.684342 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:41.813472 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.813432 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-workload-certs\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.813472 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.813477 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-credential-socket\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.813789 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.813548 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/debbff81-ed54-4548-b64d-737eeaffd528-istio-token\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.813789 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.813637 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-istio-data\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.813789 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.813673 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debbff81-ed54-4548-b64d-737eeaffd528-istiod-ca-cert\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.813789 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.813701 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-workload-socket\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.813958 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.813793 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debbff81-ed54-4548-b64d-737eeaffd528-istio-podinfo\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.813958 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.813838 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-istio-envoy\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.813958 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.813864 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wh4d\" (UniqueName: \"kubernetes.io/projected/debbff81-ed54-4548-b64d-737eeaffd528-kube-api-access-2wh4d\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.915033 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.914950 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debbff81-ed54-4548-b64d-737eeaffd528-istio-podinfo\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.915033 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.914996 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-istio-envoy\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.915306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.915117 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wh4d\" (UniqueName: \"kubernetes.io/projected/debbff81-ed54-4548-b64d-737eeaffd528-kube-api-access-2wh4d\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.915306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.915223 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-workload-certs\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.915306 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.915274 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-credential-socket\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.915472 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.915366 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/debbff81-ed54-4548-b64d-737eeaffd528-istio-token\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.915472 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.915418 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-istio-data\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.915472 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.915457 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debbff81-ed54-4548-b64d-737eeaffd528-istiod-ca-cert\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.915666 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.915501 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-workload-socket\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.915728 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.915671 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-workload-certs\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.915728 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.915707 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-credential-socket\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.915897 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.915747 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-istio-data\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.916085 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.916061 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-workload-socket\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.916329 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.916307 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debbff81-ed54-4548-b64d-737eeaffd528-istiod-ca-cert\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.917663 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.917644 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-istio-envoy\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.917792 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.917774 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debbff81-ed54-4548-b64d-737eeaffd528-istio-podinfo\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.927528 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.927497 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/debbff81-ed54-4548-b64d-737eeaffd528-istio-token\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.927741 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.927542 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wh4d\" (UniqueName: \"kubernetes.io/projected/debbff81-ed54-4548-b64d-737eeaffd528-kube-api-access-2wh4d\") pod \"e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.966550 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.966510 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:41.976951 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:41.976914 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:42.135141 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:42.135116 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9"] Jul 21 00:58:42.136558 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:58:42.136529 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddebbff81_ed54_4548_b64d_737eeaffd528.slice/crio-016fb001a8095eccc5d165df9eb4e634af3b855a50668feca3313aa5015cb320 WatchSource:0}: Error finding container 016fb001a8095eccc5d165df9eb4e634af3b855a50668feca3313aa5015cb320: Status 404 returned error can't find the container with id 016fb001a8095eccc5d165df9eb4e634af3b855a50668feca3313aa5015cb320 Jul 21 00:58:42.138739 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:42.138702 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:58:42.138846 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:42.138794 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:58:42.138910 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:42.138843 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:58:42.839831 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:42.839791 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" event={"ID":"debbff81-ed54-4548-b64d-737eeaffd528","Type":"ContainerStarted","Data":"b9dca32ccf29821d557e480369a2424314e9aed1542da84aec61687e542836e8"} Jul 21 00:58:42.840254 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:42.839866 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" event={"ID":"debbff81-ed54-4548-b64d-737eeaffd528","Type":"ContainerStarted","Data":"016fb001a8095eccc5d165df9eb4e634af3b855a50668feca3313aa5015cb320"} Jul 21 00:58:42.863160 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:42.863104 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" podStartSLOduration=1.863088328 podStartE2EDuration="1.863088328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:58:42.86154295 +0000 UTC m=+2517.076923280" watchObservedRunningTime="2026-07-21 00:58:42.863088328 +0000 UTC m=+2517.078468654" Jul 21 00:58:42.967065 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:42.967022 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:43.972633 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:43.972588 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:44.851229 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:44.851169 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:44.852082 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:44.852062 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:44.918191 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:44.918155 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:45.013546 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:45.013504 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:45.110664 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:45.110529 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:45.679196 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:45.679158 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:45.778970 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:45.778938 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:45.860579 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:45.860547 2580 generic.go:356] "Generic (PLEG): container finished" podID="9823a172-2287-484e-a10f-f2c6e27d0e19" containerID="4fb31371dc260cd32ceee0a71b88080bce4488193add1e85a8e848b26f2d5fad" exitCode=0 Jul 21 00:58:45.860808 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:45.860642 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" event={"ID":"9823a172-2287-484e-a10f-f2c6e27d0e19","Type":"ContainerDied","Data":"4fb31371dc260cd32ceee0a71b88080bce4488193add1e85a8e848b26f2d5fad"} Jul 21 00:58:45.860808 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:45.860708 2580 scope.go:117] "RemoveContainer" containerID="514d8bb4ef3a582c64b9e54a56e6cf13ac5b11f674d03f51dbf8e81c9b51c558" Jul 21 00:58:45.976186 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:45.976156 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:58:46.053748 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.053714 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-workload-certs\") pod \"9823a172-2287-484e-a10f-f2c6e27d0e19\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " Jul 21 00:58:46.054259 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.053783 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpbml\" (UniqueName: \"kubernetes.io/projected/9823a172-2287-484e-a10f-f2c6e27d0e19-kube-api-access-dpbml\") pod \"9823a172-2287-484e-a10f-f2c6e27d0e19\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " Jul 21 00:58:46.054259 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.053825 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9823a172-2287-484e-a10f-f2c6e27d0e19-istiod-ca-cert\") pod \"9823a172-2287-484e-a10f-f2c6e27d0e19\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " Jul 21 00:58:46.054259 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.053858 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-istio-envoy\") pod \"9823a172-2287-484e-a10f-f2c6e27d0e19\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " Jul 21 00:58:46.054259 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.053917 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-istio-data\") pod \"9823a172-2287-484e-a10f-f2c6e27d0e19\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " Jul 21 00:58:46.054259 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.053942 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9823a172-2287-484e-a10f-f2c6e27d0e19-istio-podinfo\") pod \"9823a172-2287-484e-a10f-f2c6e27d0e19\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " Jul 21 00:58:46.054259 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.053989 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-credential-socket\") pod \"9823a172-2287-484e-a10f-f2c6e27d0e19\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " Jul 21 00:58:46.054259 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.054011 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9823a172-2287-484e-a10f-f2c6e27d0e19" (UID: "9823a172-2287-484e-a10f-f2c6e27d0e19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:58:46.054566 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.054265 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-workload-certs\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:58:46.054566 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.054273 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9823a172-2287-484e-a10f-f2c6e27d0e19" (UID: "9823a172-2287-484e-a10f-f2c6e27d0e19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:58:46.054566 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.054314 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9823a172-2287-484e-a10f-f2c6e27d0e19-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9823a172-2287-484e-a10f-f2c6e27d0e19" (UID: "9823a172-2287-484e-a10f-f2c6e27d0e19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 00:58:46.054975 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.054949 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-istio-data" (OuterVolumeSpecName: "istio-data") pod "9823a172-2287-484e-a10f-f2c6e27d0e19" (UID: "9823a172-2287-484e-a10f-f2c6e27d0e19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:58:46.056310 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.056285 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9823a172-2287-484e-a10f-f2c6e27d0e19" (UID: "9823a172-2287-484e-a10f-f2c6e27d0e19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:58:46.056310 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.056289 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9823a172-2287-484e-a10f-f2c6e27d0e19-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9823a172-2287-484e-a10f-f2c6e27d0e19" (UID: "9823a172-2287-484e-a10f-f2c6e27d0e19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 00:58:46.056310 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.056294 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9823a172-2287-484e-a10f-f2c6e27d0e19-kube-api-access-dpbml" (OuterVolumeSpecName: "kube-api-access-dpbml") pod "9823a172-2287-484e-a10f-f2c6e27d0e19" (UID: "9823a172-2287-484e-a10f-f2c6e27d0e19"). InnerVolumeSpecName "kube-api-access-dpbml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:58:46.154933 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.154901 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9823a172-2287-484e-a10f-f2c6e27d0e19-istio-token\") pod \"9823a172-2287-484e-a10f-f2c6e27d0e19\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " Jul 21 00:58:46.154933 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.154941 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-workload-socket\") pod \"9823a172-2287-484e-a10f-f2c6e27d0e19\" (UID: \"9823a172-2287-484e-a10f-f2c6e27d0e19\") " Jul 21 00:58:46.155151 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.155095 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpbml\" (UniqueName: \"kubernetes.io/projected/9823a172-2287-484e-a10f-f2c6e27d0e19-kube-api-access-dpbml\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:58:46.155151 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.155105 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9823a172-2287-484e-a10f-f2c6e27d0e19-istiod-ca-cert\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:58:46.155151 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.155114 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-istio-envoy\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:58:46.155151 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.155123 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-istio-data\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:58:46.155151 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.155133 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9823a172-2287-484e-a10f-f2c6e27d0e19-istio-podinfo\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:58:46.155151 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.155141 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-credential-socket\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:58:46.155359 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.155212 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9823a172-2287-484e-a10f-f2c6e27d0e19" (UID: "9823a172-2287-484e-a10f-f2c6e27d0e19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 00:58:46.157122 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.157096 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9823a172-2287-484e-a10f-f2c6e27d0e19-istio-token" (OuterVolumeSpecName: "istio-token") pod "9823a172-2287-484e-a10f-f2c6e27d0e19" (UID: "9823a172-2287-484e-a10f-f2c6e27d0e19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 00:58:46.255990 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.255946 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9823a172-2287-484e-a10f-f2c6e27d0e19-istio-token\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:58:46.255990 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.255973 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9823a172-2287-484e-a10f-f2c6e27d0e19-workload-socket\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 00:58:46.869405 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.869374 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" Jul 21 00:58:46.869651 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.869406 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns" event={"ID":"9823a172-2287-484e-a10f-f2c6e27d0e19","Type":"ContainerDied","Data":"ab446cfa8bb56e3a31be6e25dc0db9c572cab4c111ae03a2aad10ec6279c34f4"} Jul 21 00:58:46.869651 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.869498 2580 scope.go:117] "RemoveContainer" containerID="4fb31371dc260cd32ceee0a71b88080bce4488193add1e85a8e848b26f2d5fad" Jul 21 00:58:46.888114 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.888081 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns"] Jul 21 00:58:46.893710 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:46.893676 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a3348a-openshift-default-68dcc65977-m4sns"] Jul 21 00:58:47.663682 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:47.663644 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:47.715071 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:47.715034 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:48.298313 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:48.298271 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9823a172-2287-484e-a10f-f2c6e27d0e19" path="/var/lib/kubelet/pods/9823a172-2287-484e-a10f-f2c6e27d0e19/volumes" Jul 21 00:58:49.329272 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:49.329241 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:49.684862 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:49.684778 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:49.975833 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:49.975723 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g"] Jul 21 00:58:49.976271 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:49.976249 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9823a172-2287-484e-a10f-f2c6e27d0e19" containerName="istio-proxy" Jul 21 00:58:49.976377 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:49.976273 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9823a172-2287-484e-a10f-f2c6e27d0e19" containerName="istio-proxy" Jul 21 00:58:49.976436 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:49.976397 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9823a172-2287-484e-a10f-f2c6e27d0e19" containerName="istio-proxy" Jul 21 00:58:49.976436 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:49.976410 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9823a172-2287-484e-a10f-f2c6e27d0e19" containerName="istio-proxy" Jul 21 00:58:49.981512 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:49.981486 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:49.984384 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:49.984363 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-b4ed4b-openshift-default-dockercfg-k9v96\"" Jul 21 00:58:49.990904 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:49.990874 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr74q\" (UniqueName: \"kubernetes.io/projected/0c5b99bc-61b0-4917-9df1-c52583bb008d-kube-api-access-kr74q\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:49.991031 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:49.990919 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c5b99bc-61b0-4917-9df1-c52583bb008d-istio-token\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:49.991031 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:49.990949 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-workload-certs\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:49.991151 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:49.991054 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c5b99bc-61b0-4917-9df1-c52583bb008d-istiod-ca-cert\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:49.991151 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:49.991085 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-credential-socket\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:49.991151 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:49.991112 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-workload-socket\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:49.991315 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:49.991190 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c5b99bc-61b0-4917-9df1-c52583bb008d-istio-podinfo\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:49.991315 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:49.991222 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-istio-data\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:49.991315 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:49.991256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-istio-envoy\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:49.991959 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:49.991938 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g"] Jul 21 00:58:50.004213 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.004187 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:50.092566 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.092529 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-istio-data\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:50.092789 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.092587 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-istio-envoy\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:50.092789 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.092671 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr74q\" (UniqueName: \"kubernetes.io/projected/0c5b99bc-61b0-4917-9df1-c52583bb008d-kube-api-access-kr74q\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:50.092789 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.092698 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c5b99bc-61b0-4917-9df1-c52583bb008d-istio-token\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:50.092789 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.092727 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-workload-certs\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:50.093058 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.092812 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c5b99bc-61b0-4917-9df1-c52583bb008d-istiod-ca-cert\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:50.093058 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.092843 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-credential-socket\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:50.093058 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.092871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-workload-socket\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:50.093058 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.092949 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c5b99bc-61b0-4917-9df1-c52583bb008d-istio-podinfo\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:50.093642 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.093546 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-credential-socket\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:50.093905 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.093856 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-workload-socket\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:50.093905 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.093881 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c5b99bc-61b0-4917-9df1-c52583bb008d-istiod-ca-cert\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:50.094088 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.093974 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-workload-certs\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:50.094148 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.094093 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-istio-data\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:50.096220 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.096193 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-istio-envoy\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:50.096506 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.096488 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c5b99bc-61b0-4917-9df1-c52583bb008d-istio-podinfo\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:50.102403 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.101925 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c5b99bc-61b0-4917-9df1-c52583bb008d-istio-token\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:50.102403 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.102115 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr74q\" (UniqueName: \"kubernetes.io/projected/0c5b99bc-61b0-4917-9df1-c52583bb008d-kube-api-access-kr74q\") pod \"e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:50.131425 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.131375 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:50.294977 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.294881 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:50.464701 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.464667 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g"] Jul 21 00:58:50.466831 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:58:50.466804 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c5b99bc_61b0_4917_9df1_c52583bb008d.slice/crio-73e6abe6fc0279aa553ef2b2ca8e925aeec89e67d9b015c961362133971230d3 WatchSource:0}: Error finding container 73e6abe6fc0279aa553ef2b2ca8e925aeec89e67d9b015c961362133971230d3: Status 404 returned error can't find the container with id 73e6abe6fc0279aa553ef2b2ca8e925aeec89e67d9b015c961362133971230d3 Jul 21 00:58:50.469110 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.469075 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:58:50.469200 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.469146 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:58:50.469200 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.469177 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:58:50.897923 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.897879 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" event={"ID":"0c5b99bc-61b0-4917-9df1-c52583bb008d","Type":"ContainerStarted","Data":"bc595f3d7489364df6ef6ea7af682e86050d11c2ca3674efe4d9f6bc825a9845"} Jul 21 00:58:50.898126 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.897949 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" event={"ID":"0c5b99bc-61b0-4917-9df1-c52583bb008d","Type":"ContainerStarted","Data":"73e6abe6fc0279aa553ef2b2ca8e925aeec89e67d9b015c961362133971230d3"} Jul 21 00:58:50.920998 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:50.920941 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" podStartSLOduration=1.920921493 podStartE2EDuration="1.920921493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:58:50.91776811 +0000 UTC m=+2525.133148436" watchObservedRunningTime="2026-07-21 00:58:50.920921493 +0000 UTC m=+2525.136301853" Jul 21 00:58:51.297388 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:51.295398 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:51.312275 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:51.312227 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:51.908708 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:51.908667 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:51.914511 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:51.914468 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:58:52.873175 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:52.873142 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:52.974391 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:52.974353 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:53.050177 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:53.050141 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:53.162394 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:53.162291 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:53.572653 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:53.572617 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:53.673519 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:53.673482 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:58.401542 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:58.401447 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:58:58.956807 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:58.956775 2580 generic.go:356] "Generic (PLEG): container finished" podID="debbff81-ed54-4548-b64d-737eeaffd528" containerID="b9dca32ccf29821d557e480369a2424314e9aed1542da84aec61687e542836e8" exitCode=0 Jul 21 00:58:58.956998 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:58.956828 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" event={"ID":"debbff81-ed54-4548-b64d-737eeaffd528","Type":"ContainerDied","Data":"b9dca32ccf29821d557e480369a2424314e9aed1542da84aec61687e542836e8"} Jul 21 00:58:58.957232 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:58.957217 2580 scope.go:117] "RemoveContainer" containerID="b9dca32ccf29821d557e480369a2424314e9aed1542da84aec61687e542836e8" Jul 21 00:58:58.958366 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:58.958333 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:58:58.958480 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:58.958425 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:58:58.958544 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:58.958476 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:58:59.851307 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:59.851251 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:58:59.964795 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:59.964756 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" event={"ID":"debbff81-ed54-4548-b64d-737eeaffd528","Type":"ContainerStarted","Data":"821331ac6cd07971ad16f933831919a5d43a683dde9b71e55c1db5e257303c6e"} Jul 21 00:58:59.967117 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:58:59.967092 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:59:00.968156 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:00.968120 2580 patch_prober.go:28] interesting pod/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Jul 21 00:59:00.968570 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:00.968182 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" podUID="debbff81-ed54-4548-b64d-737eeaffd528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Jul 21 00:59:01.967076 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:01.967037 2580 patch_prober.go:28] interesting pod/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Jul 21 00:59:01.967289 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:01.967128 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" podUID="debbff81-ed54-4548-b64d-737eeaffd528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Jul 21 00:59:02.967508 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:02.967460 2580 patch_prober.go:28] interesting pod/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Jul 21 00:59:02.967934 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:02.967537 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" podUID="debbff81-ed54-4548-b64d-737eeaffd528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Jul 21 00:59:03.971936 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:03.971899 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:59:03.972461 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:03.972339 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:59:03.973069 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:03.973046 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 00:59:05.000378 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:05.000347 2580 generic.go:356] "Generic (PLEG): container finished" podID="0c5b99bc-61b0-4917-9df1-c52583bb008d" containerID="bc595f3d7489364df6ef6ea7af682e86050d11c2ca3674efe4d9f6bc825a9845" exitCode=0 Jul 21 00:59:05.000850 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:05.000423 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" event={"ID":"0c5b99bc-61b0-4917-9df1-c52583bb008d","Type":"ContainerDied","Data":"bc595f3d7489364df6ef6ea7af682e86050d11c2ca3674efe4d9f6bc825a9845"} Jul 21 00:59:05.000901 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:05.000884 2580 scope.go:117] "RemoveContainer" containerID="bc595f3d7489364df6ef6ea7af682e86050d11c2ca3674efe4d9f6bc825a9845" Jul 21 00:59:05.002072 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:05.002044 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:59:05.002136 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:05.002118 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:59:05.002172 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:05.002159 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 21 00:59:06.018779 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:06.018734 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" event={"ID":"0c5b99bc-61b0-4917-9df1-c52583bb008d","Type":"ContainerStarted","Data":"d782b8ea22758dbbf3e0975dd83d4c4fde9c83d2d04acdbf562a03e3ffea62f4"} Jul 21 00:59:06.299820 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:06.299385 2580 patch_prober.go:28] interesting pod/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Jul 21 00:59:06.299820 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:06.299452 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" podUID="0c5b99bc-61b0-4917-9df1-c52583bb008d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Jul 21 00:59:06.305378 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:06.305332 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:59:06.648840 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:06.648805 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:59:06.749736 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:06.749701 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:59:06.912946 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:06.912844 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:59:07.299824 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:07.299718 2580 patch_prober.go:28] interesting pod/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Jul 21 00:59:07.300443 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:07.300403 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" podUID="0c5b99bc-61b0-4917-9df1-c52583bb008d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Jul 21 00:59:08.295296 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:08.295258 2580 patch_prober.go:28] interesting pod/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Jul 21 00:59:08.295661 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:08.295631 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" podUID="0c5b99bc-61b0-4917-9df1-c52583bb008d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Jul 21 00:59:09.295543 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:09.295503 2580 patch_prober.go:28] interesting pod/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Jul 21 00:59:09.296196 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:09.296161 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" podUID="0c5b99bc-61b0-4917-9df1-c52583bb008d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Jul 21 00:59:09.906679 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:09.906640 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:59:10.299532 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:10.299457 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:59:10.300545 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:10.300525 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 00:59:14.351740 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:14.351696 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:59:14.454340 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:14.454306 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:59:35.271519 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:35.271480 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:59:35.370548 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:35.370515 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:59:42.227994 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.227949 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh"] Jul 21 00:59:42.228481 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.228465 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9823a172-2287-484e-a10f-f2c6e27d0e19" containerName="istio-proxy" Jul 21 00:59:42.228531 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.228483 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9823a172-2287-484e-a10f-f2c6e27d0e19" containerName="istio-proxy" Jul 21 00:59:42.231915 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.231896 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 00:59:42.235564 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.235542 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b4ed4b\"/\"default-dockercfg-kr9sp\"" Jul 21 00:59:42.235749 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.235545 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b4ed4b\"/\"kube-root-ca.crt\"" Jul 21 00:59:42.235835 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.235780 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b4ed4b\"/\"auth-test-model-b4ed4b-kserve-self-signed-certs\"" Jul 21 00:59:42.235835 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.235790 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b4ed4b\"/\"openshift-service-ca.crt\"" Jul 21 00:59:42.242747 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.242715 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh"] Jul 21 00:59:42.330965 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.330924 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f3c5194-4610-4a6f-95c4-2fad26720493-home\") pod \"auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh\" (UID: \"7f3c5194-4610-4a6f-95c4-2fad26720493\") " pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 00:59:42.331174 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.331059 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f3c5194-4610-4a6f-95c4-2fad26720493-dshm\") pod \"auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh\" (UID: \"7f3c5194-4610-4a6f-95c4-2fad26720493\") " pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 00:59:42.331174 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.331128 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f3c5194-4610-4a6f-95c4-2fad26720493-model-cache\") pod \"auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh\" (UID: \"7f3c5194-4610-4a6f-95c4-2fad26720493\") " pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 00:59:42.331313 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.331224 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgpq6\" (UniqueName: \"kubernetes.io/projected/7f3c5194-4610-4a6f-95c4-2fad26720493-kube-api-access-jgpq6\") pod \"auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh\" (UID: \"7f3c5194-4610-4a6f-95c4-2fad26720493\") " pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 00:59:42.331374 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.331353 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3c5194-4610-4a6f-95c4-2fad26720493-tmp-dir\") pod \"auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh\" (UID: \"7f3c5194-4610-4a6f-95c4-2fad26720493\") " pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 00:59:42.331469 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.331450 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f3c5194-4610-4a6f-95c4-2fad26720493-tls-certs\") pod \"auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh\" (UID: \"7f3c5194-4610-4a6f-95c4-2fad26720493\") " pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 00:59:42.346496 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.346418 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 00:59:42.432511 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.432471 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f3c5194-4610-4a6f-95c4-2fad26720493-dshm\") pod \"auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh\" (UID: \"7f3c5194-4610-4a6f-95c4-2fad26720493\") " pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 00:59:42.432511 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.432526 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f3c5194-4610-4a6f-95c4-2fad26720493-model-cache\") pod \"auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh\" (UID: \"7f3c5194-4610-4a6f-95c4-2fad26720493\") " pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 00:59:42.432825 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.432652 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgpq6\" (UniqueName: \"kubernetes.io/projected/7f3c5194-4610-4a6f-95c4-2fad26720493-kube-api-access-jgpq6\") pod \"auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh\" (UID: \"7f3c5194-4610-4a6f-95c4-2fad26720493\") " pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 00:59:42.432825 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.432701 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3c5194-4610-4a6f-95c4-2fad26720493-tmp-dir\") pod \"auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh\" (UID: \"7f3c5194-4610-4a6f-95c4-2fad26720493\") " pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 00:59:42.432825 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.432756 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f3c5194-4610-4a6f-95c4-2fad26720493-tls-certs\") pod \"auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh\" (UID: \"7f3c5194-4610-4a6f-95c4-2fad26720493\") " pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 00:59:42.432825 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.432795 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f3c5194-4610-4a6f-95c4-2fad26720493-home\") pod \"auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh\" (UID: \"7f3c5194-4610-4a6f-95c4-2fad26720493\") " pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 00:59:42.433068 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.432963 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f3c5194-4610-4a6f-95c4-2fad26720493-model-cache\") pod \"auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh\" (UID: \"7f3c5194-4610-4a6f-95c4-2fad26720493\") " pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 00:59:42.433136 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.433087 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3c5194-4610-4a6f-95c4-2fad26720493-tmp-dir\") pod \"auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh\" (UID: \"7f3c5194-4610-4a6f-95c4-2fad26720493\") " pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 00:59:42.433177 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.433130 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f3c5194-4610-4a6f-95c4-2fad26720493-home\") pod \"auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh\" (UID: \"7f3c5194-4610-4a6f-95c4-2fad26720493\") " pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 00:59:42.435130 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.435104 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f3c5194-4610-4a6f-95c4-2fad26720493-dshm\") pod \"auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh\" (UID: \"7f3c5194-4610-4a6f-95c4-2fad26720493\") " pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 00:59:42.435299 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.435280 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f3c5194-4610-4a6f-95c4-2fad26720493-tls-certs\") pod \"auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh\" (UID: \"7f3c5194-4610-4a6f-95c4-2fad26720493\") " pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 00:59:42.443361 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.443335 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgpq6\" (UniqueName: \"kubernetes.io/projected/7f3c5194-4610-4a6f-95c4-2fad26720493-kube-api-access-jgpq6\") pod \"auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh\" (UID: \"7f3c5194-4610-4a6f-95c4-2fad26720493\") " pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 00:59:42.543761 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.543652 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 00:59:42.694953 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:42.694925 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh"] Jul 21 00:59:42.696799 ip-10-0-139-117 kubenswrapper[2580]: W0721 00:59:42.696769 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f3c5194_4610_4a6f_95c4_2fad26720493.slice/crio-37e446c49aa8a804560b83bcf35188f6345f3111ea0b90025637af7d17f7148a WatchSource:0}: Error finding container 37e446c49aa8a804560b83bcf35188f6345f3111ea0b90025637af7d17f7148a: Status 404 returned error can't find the container with id 37e446c49aa8a804560b83bcf35188f6345f3111ea0b90025637af7d17f7148a Jul 21 00:59:43.250653 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:43.250581 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" event={"ID":"7f3c5194-4610-4a6f-95c4-2fad26720493","Type":"ContainerStarted","Data":"23006d854b45e049306003e5f62ee328b1a20669391b553d1a9055ac02a2074b"} Jul 21 00:59:43.251125 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:43.250664 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" event={"ID":"7f3c5194-4610-4a6f-95c4-2fad26720493","Type":"ContainerStarted","Data":"37e446c49aa8a804560b83bcf35188f6345f3111ea0b90025637af7d17f7148a"} Jul 21 00:59:43.266939 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:43.266879 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" podStartSLOduration=1.266861947 podStartE2EDuration="1.266861947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 00:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 00:59:43.266258184 +0000 UTC m=+2577.481638511" watchObservedRunningTime="2026-07-21 00:59:43.266861947 +0000 UTC m=+2577.482242273" Jul 21 00:59:52.544386 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:52.544339 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 00:59:52.544941 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:52.544412 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 00:59:52.556962 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:52.556929 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 00:59:53.325082 ip-10-0-139-117 kubenswrapper[2580]: I0721 00:59:53.325052 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 01:01:09.764395 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:01:09.764352 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:01:43.957887 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:01:43.957789 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:01:44.870414 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:01:44.870374 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh"] Jul 21 01:01:44.892670 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:01:44.892626 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:01:54.577729 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:01:54.577688 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:01:59.895506 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:01:59.895459 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" podUID="7f3c5194-4610-4a6f-95c4-2fad26720493" containerName="main" containerID="cri-o://23006d854b45e049306003e5f62ee328b1a20669391b553d1a9055ac02a2074b" gracePeriod=45 Jul 21 01:02:00.080232 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:00.080193 2580 generic.go:356] "Generic (PLEG): container finished" podID="7f3c5194-4610-4a6f-95c4-2fad26720493" containerID="23006d854b45e049306003e5f62ee328b1a20669391b553d1a9055ac02a2074b" exitCode=0 Jul 21 01:02:00.080397 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:00.080266 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" event={"ID":"7f3c5194-4610-4a6f-95c4-2fad26720493","Type":"ContainerDied","Data":"23006d854b45e049306003e5f62ee328b1a20669391b553d1a9055ac02a2074b"} Jul 21 01:02:00.151405 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:00.151330 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 01:02:00.219282 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:00.219240 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f3c5194-4610-4a6f-95c4-2fad26720493-home\") pod \"7f3c5194-4610-4a6f-95c4-2fad26720493\" (UID: \"7f3c5194-4610-4a6f-95c4-2fad26720493\") " Jul 21 01:02:00.219282 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:00.219285 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f3c5194-4610-4a6f-95c4-2fad26720493-model-cache\") pod \"7f3c5194-4610-4a6f-95c4-2fad26720493\" (UID: \"7f3c5194-4610-4a6f-95c4-2fad26720493\") " Jul 21 01:02:00.219530 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:00.219327 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f3c5194-4610-4a6f-95c4-2fad26720493-tls-certs\") pod \"7f3c5194-4610-4a6f-95c4-2fad26720493\" (UID: \"7f3c5194-4610-4a6f-95c4-2fad26720493\") " Jul 21 01:02:00.219530 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:00.219361 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3c5194-4610-4a6f-95c4-2fad26720493-tmp-dir\") pod \"7f3c5194-4610-4a6f-95c4-2fad26720493\" (UID: \"7f3c5194-4610-4a6f-95c4-2fad26720493\") " Jul 21 01:02:00.219530 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:00.219391 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f3c5194-4610-4a6f-95c4-2fad26720493-dshm\") pod \"7f3c5194-4610-4a6f-95c4-2fad26720493\" (UID: \"7f3c5194-4610-4a6f-95c4-2fad26720493\") " Jul 21 01:02:00.219530 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:00.219481 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgpq6\" (UniqueName: \"kubernetes.io/projected/7f3c5194-4610-4a6f-95c4-2fad26720493-kube-api-access-jgpq6\") pod \"7f3c5194-4610-4a6f-95c4-2fad26720493\" (UID: \"7f3c5194-4610-4a6f-95c4-2fad26720493\") " Jul 21 01:02:00.219786 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:00.219510 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3c5194-4610-4a6f-95c4-2fad26720493-home" (OuterVolumeSpecName: "home") pod "7f3c5194-4610-4a6f-95c4-2fad26720493" (UID: "7f3c5194-4610-4a6f-95c4-2fad26720493"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:02:00.219786 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:00.219634 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3c5194-4610-4a6f-95c4-2fad26720493-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f3c5194-4610-4a6f-95c4-2fad26720493" (UID: "7f3c5194-4610-4a6f-95c4-2fad26720493"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:02:00.219786 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:00.219680 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3c5194-4610-4a6f-95c4-2fad26720493-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f3c5194-4610-4a6f-95c4-2fad26720493" (UID: "7f3c5194-4610-4a6f-95c4-2fad26720493"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:02:00.219932 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:00.219884 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f3c5194-4610-4a6f-95c4-2fad26720493-home\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:02:00.219932 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:00.219898 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f3c5194-4610-4a6f-95c4-2fad26720493-model-cache\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:02:00.219932 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:00.219907 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f3c5194-4610-4a6f-95c4-2fad26720493-tmp-dir\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:02:00.222157 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:00.222125 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3c5194-4610-4a6f-95c4-2fad26720493-kube-api-access-jgpq6" (OuterVolumeSpecName: "kube-api-access-jgpq6") pod "7f3c5194-4610-4a6f-95c4-2fad26720493" (UID: "7f3c5194-4610-4a6f-95c4-2fad26720493"). InnerVolumeSpecName "kube-api-access-jgpq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:02:00.222157 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:00.222134 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f3c5194-4610-4a6f-95c4-2fad26720493-dshm" (OuterVolumeSpecName: "dshm") pod "7f3c5194-4610-4a6f-95c4-2fad26720493" (UID: "7f3c5194-4610-4a6f-95c4-2fad26720493"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:02:00.222326 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:00.222202 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f3c5194-4610-4a6f-95c4-2fad26720493-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f3c5194-4610-4a6f-95c4-2fad26720493" (UID: "7f3c5194-4610-4a6f-95c4-2fad26720493"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 01:02:00.320982 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:00.320938 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f3c5194-4610-4a6f-95c4-2fad26720493-dshm\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:02:00.320982 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:00.320974 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgpq6\" (UniqueName: \"kubernetes.io/projected/7f3c5194-4610-4a6f-95c4-2fad26720493-kube-api-access-jgpq6\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:02:00.320982 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:00.320984 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f3c5194-4610-4a6f-95c4-2fad26720493-tls-certs\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:02:01.087793 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:01.087752 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" event={"ID":"7f3c5194-4610-4a6f-95c4-2fad26720493","Type":"ContainerDied","Data":"37e446c49aa8a804560b83bcf35188f6345f3111ea0b90025637af7d17f7148a"} Jul 21 01:02:01.087793 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:01.087775 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh" Jul 21 01:02:01.088350 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:01.087812 2580 scope.go:117] "RemoveContainer" containerID="23006d854b45e049306003e5f62ee328b1a20669391b553d1a9055ac02a2074b" Jul 21 01:02:01.105566 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:01.105524 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh"] Jul 21 01:02:01.108908 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:01.108871 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b4ed4b/auth-test-model-b4ed4b-kserve-cfbb45cc9-9mfzh"] Jul 21 01:02:02.299230 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:02.299191 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f3c5194-4610-4a6f-95c4-2fad26720493" path="/var/lib/kubelet/pods/7f3c5194-4610-4a6f-95c4-2fad26720493/volumes" Jul 21 01:02:23.248827 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:02:23.248789 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:03:09.757988 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:03:09.757905 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:03:10.682837 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:03:10.682802 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:03:20.543250 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:03:20.543215 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:03:33.533809 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:03:33.533773 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:03:51.695836 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:03:51.695800 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:03:51.797584 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:03:51.797547 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:04:04.276974 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:04.276898 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:04:07.416955 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:07.416916 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:04:07.517260 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:07.517219 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:04:15.117478 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:15.117448 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:04:15.217785 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:15.217743 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:04:36.023943 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:36.023902 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:04:36.132782 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:36.132746 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:04:45.767313 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:45.767221 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:04:46.653718 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:46.653687 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:04:53.622921 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:53.622884 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7191-140002104607056"] Jul 21 01:04:53.623325 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:53.623301 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f3c5194-4610-4a6f-95c4-2fad26720493" containerName="main" Jul 21 01:04:53.623325 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:53.623311 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f3c5194-4610-4a6f-95c4-2fad26720493" containerName="main" Jul 21 01:04:53.623404 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:53.623386 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f3c5194-4610-4a6f-95c4-2fad26720493" containerName="main" Jul 21 01:04:53.626500 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:53.626479 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7191-140002104607056" Jul 21 01:04:53.629648 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:53.629591 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4xsqr\"" Jul 21 01:04:53.632020 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:53.631994 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7191-140002104607056"] Jul 21 01:04:53.723272 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:53.723231 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2cjj\" (UniqueName: \"kubernetes.io/projected/ef8f581b-6141-4137-86f5-2ebd16bb9583-kube-api-access-g2cjj\") pod \"test-curl-7191-140002104607056\" (UID: \"ef8f581b-6141-4137-86f5-2ebd16bb9583\") " pod="opendatahub/test-curl-7191-140002104607056" Jul 21 01:04:53.824449 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:53.824400 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2cjj\" (UniqueName: \"kubernetes.io/projected/ef8f581b-6141-4137-86f5-2ebd16bb9583-kube-api-access-g2cjj\") pod \"test-curl-7191-140002104607056\" (UID: \"ef8f581b-6141-4137-86f5-2ebd16bb9583\") " pod="opendatahub/test-curl-7191-140002104607056" Jul 21 01:04:53.832771 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:53.832743 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2cjj\" (UniqueName: \"kubernetes.io/projected/ef8f581b-6141-4137-86f5-2ebd16bb9583-kube-api-access-g2cjj\") pod \"test-curl-7191-140002104607056\" (UID: \"ef8f581b-6141-4137-86f5-2ebd16bb9583\") " pod="opendatahub/test-curl-7191-140002104607056" Jul 21 01:04:53.938971 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:53.938878 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7191-140002104607056" Jul 21 01:04:54.086337 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:54.086309 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7191-140002104607056"] Jul 21 01:04:54.088200 ip-10-0-139-117 kubenswrapper[2580]: W0721 01:04:54.088169 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef8f581b_6141_4137_86f5_2ebd16bb9583.slice/crio-eeaccd6c0108db4280db9c8176b719a719ff9bc077b623aae25c5622cfb7c81e WatchSource:0}: Error finding container eeaccd6c0108db4280db9c8176b719a719ff9bc077b623aae25c5622cfb7c81e: Status 404 returned error can't find the container with id eeaccd6c0108db4280db9c8176b719a719ff9bc077b623aae25c5622cfb7c81e Jul 21 01:04:54.089666 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:54.089648 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 01:04:54.125912 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:54.125883 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7191-140002104607056" event={"ID":"ef8f581b-6141-4137-86f5-2ebd16bb9583","Type":"ContainerStarted","Data":"eeaccd6c0108db4280db9c8176b719a719ff9bc077b623aae25c5622cfb7c81e"} Jul 21 01:04:55.133778 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:55.133729 2580 generic.go:356] "Generic (PLEG): container finished" podID="ef8f581b-6141-4137-86f5-2ebd16bb9583" containerID="5abcff4e2952bb0ddead0a3304a4722cbeacf2cd7b831ae45cdf6d6525da0c7f" exitCode=0 Jul 21 01:04:55.134201 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:55.133809 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7191-140002104607056" event={"ID":"ef8f581b-6141-4137-86f5-2ebd16bb9583","Type":"ContainerDied","Data":"5abcff4e2952bb0ddead0a3304a4722cbeacf2cd7b831ae45cdf6d6525da0c7f"} Jul 21 01:04:56.287876 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:56.287845 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7191-140002104607056" Jul 21 01:04:56.308698 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:56.308669 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7191-140002104607056_ef8f581b-6141-4137-86f5-2ebd16bb9583/test-curl-7191-140002104607056/0.log" Jul 21 01:04:56.334496 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:56.334458 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7191-140002104607056"] Jul 21 01:04:56.338279 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:56.338242 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7191-140002104607056"] Jul 21 01:04:56.348802 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:56.348761 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2cjj\" (UniqueName: \"kubernetes.io/projected/ef8f581b-6141-4137-86f5-2ebd16bb9583-kube-api-access-g2cjj\") pod \"ef8f581b-6141-4137-86f5-2ebd16bb9583\" (UID: \"ef8f581b-6141-4137-86f5-2ebd16bb9583\") " Jul 21 01:04:56.351135 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:56.351102 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8f581b-6141-4137-86f5-2ebd16bb9583-kube-api-access-g2cjj" (OuterVolumeSpecName: "kube-api-access-g2cjj") pod "ef8f581b-6141-4137-86f5-2ebd16bb9583" (UID: "ef8f581b-6141-4137-86f5-2ebd16bb9583"). InnerVolumeSpecName "kube-api-access-g2cjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:04:56.450239 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:56.450139 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2cjj\" (UniqueName: \"kubernetes.io/projected/ef8f581b-6141-4137-86f5-2ebd16bb9583-kube-api-access-g2cjj\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:04:56.469845 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:56.469804 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7191-140002104607056"] Jul 21 01:04:56.470231 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:56.470218 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef8f581b-6141-4137-86f5-2ebd16bb9583" containerName="test-curl-7191-140002104607056" Jul 21 01:04:56.470282 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:56.470233 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef8f581b-6141-4137-86f5-2ebd16bb9583" containerName="test-curl-7191-140002104607056" Jul 21 01:04:56.470321 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:56.470312 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef8f581b-6141-4137-86f5-2ebd16bb9583" containerName="test-curl-7191-140002104607056" Jul 21 01:04:56.473423 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:56.473402 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7191-140002104607056" Jul 21 01:04:56.480399 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:56.480316 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7191-140002104607056"] Jul 21 01:04:56.551369 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:56.551328 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ldzg\" (UniqueName: \"kubernetes.io/projected/554e14d0-bd94-481e-b7cf-c2c8a3872d01-kube-api-access-7ldzg\") pod \"test-curl-7191-140002104607056\" (UID: \"554e14d0-bd94-481e-b7cf-c2c8a3872d01\") " pod="opendatahub/test-curl-7191-140002104607056" Jul 21 01:04:56.652169 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:56.652128 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ldzg\" (UniqueName: \"kubernetes.io/projected/554e14d0-bd94-481e-b7cf-c2c8a3872d01-kube-api-access-7ldzg\") pod \"test-curl-7191-140002104607056\" (UID: \"554e14d0-bd94-481e-b7cf-c2c8a3872d01\") " pod="opendatahub/test-curl-7191-140002104607056" Jul 21 01:04:56.660996 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:56.660962 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ldzg\" (UniqueName: \"kubernetes.io/projected/554e14d0-bd94-481e-b7cf-c2c8a3872d01-kube-api-access-7ldzg\") pod \"test-curl-7191-140002104607056\" (UID: \"554e14d0-bd94-481e-b7cf-c2c8a3872d01\") " pod="opendatahub/test-curl-7191-140002104607056" Jul 21 01:04:56.784905 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:56.784810 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7191-140002104607056" Jul 21 01:04:56.933696 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:56.933670 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7191-140002104607056"] Jul 21 01:04:56.935246 ip-10-0-139-117 kubenswrapper[2580]: W0721 01:04:56.935216 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod554e14d0_bd94_481e_b7cf_c2c8a3872d01.slice/crio-5a20cb8c46f8dba214e376b194beadd667e23ad534c08a40e45c34439bfaafa2 WatchSource:0}: Error finding container 5a20cb8c46f8dba214e376b194beadd667e23ad534c08a40e45c34439bfaafa2: Status 404 returned error can't find the container with id 5a20cb8c46f8dba214e376b194beadd667e23ad534c08a40e45c34439bfaafa2 Jul 21 01:04:57.150468 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:57.150427 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7191-140002104607056" event={"ID":"554e14d0-bd94-481e-b7cf-c2c8a3872d01","Type":"ContainerStarted","Data":"5a20cb8c46f8dba214e376b194beadd667e23ad534c08a40e45c34439bfaafa2"} Jul 21 01:04:57.151809 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:57.151787 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eeaccd6c0108db4280db9c8176b719a719ff9bc077b623aae25c5622cfb7c81e" Jul 21 01:04:57.151809 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:57.151801 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7191-140002104607056" Jul 21 01:04:57.163769 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:57.163726 2580 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7191-140002104607056" oldPodUID="ef8f581b-6141-4137-86f5-2ebd16bb9583" podUID="554e14d0-bd94-481e-b7cf-c2c8a3872d01" Jul 21 01:04:58.158849 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:58.158812 2580 generic.go:356] "Generic (PLEG): container finished" podID="554e14d0-bd94-481e-b7cf-c2c8a3872d01" containerID="c1f86a7f5eccb07055c09f34d125c7ada27f1bc20b97a0dc882d0265896e7118" exitCode=0 Jul 21 01:04:58.159340 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:58.158915 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7191-140002104607056" event={"ID":"554e14d0-bd94-481e-b7cf-c2c8a3872d01","Type":"ContainerDied","Data":"c1f86a7f5eccb07055c09f34d125c7ada27f1bc20b97a0dc882d0265896e7118"} Jul 21 01:04:58.298828 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:58.298794 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef8f581b-6141-4137-86f5-2ebd16bb9583" path="/var/lib/kubelet/pods/ef8f581b-6141-4137-86f5-2ebd16bb9583/volumes" Jul 21 01:04:59.310821 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:59.310787 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7191-140002104607056" Jul 21 01:04:59.330805 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:59.330778 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7191-140002104607056_554e14d0-bd94-481e-b7cf-c2c8a3872d01/test-curl-7191-140002104607056/0.log" Jul 21 01:04:59.354545 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:59.354511 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7191-140002104607056"] Jul 21 01:04:59.360366 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:59.360334 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7191-140002104607056"] Jul 21 01:04:59.379825 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:59.379797 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ldzg\" (UniqueName: \"kubernetes.io/projected/554e14d0-bd94-481e-b7cf-c2c8a3872d01-kube-api-access-7ldzg\") pod \"554e14d0-bd94-481e-b7cf-c2c8a3872d01\" (UID: \"554e14d0-bd94-481e-b7cf-c2c8a3872d01\") " Jul 21 01:04:59.382071 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:59.382042 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/554e14d0-bd94-481e-b7cf-c2c8a3872d01-kube-api-access-7ldzg" (OuterVolumeSpecName: "kube-api-access-7ldzg") pod "554e14d0-bd94-481e-b7cf-c2c8a3872d01" (UID: "554e14d0-bd94-481e-b7cf-c2c8a3872d01"). InnerVolumeSpecName "kube-api-access-7ldzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:04:59.480793 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:04:59.480698 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ldzg\" (UniqueName: \"kubernetes.io/projected/554e14d0-bd94-481e-b7cf-c2c8a3872d01-kube-api-access-7ldzg\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:05:00.173261 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:00.173228 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7191-140002104607056" Jul 21 01:05:00.173261 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:00.173259 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a20cb8c46f8dba214e376b194beadd667e23ad534c08a40e45c34439bfaafa2" Jul 21 01:05:00.187097 ip-10-0-139-117 kubenswrapper[2580]: E0721 01:05:00.187061 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7191-140002104607056\" is forbidden: User \"system:node:ip-10-0-139-117.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-117.ec2.internal' and this object" podUID="554e14d0-bd94-481e-b7cf-c2c8a3872d01" pod="opendatahub/test-curl-7191-140002104607056" Jul 21 01:05:00.297990 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:00.297954 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="554e14d0-bd94-481e-b7cf-c2c8a3872d01" path="/var/lib/kubelet/pods/554e14d0-bd94-481e-b7cf-c2c8a3872d01/volumes" Jul 21 01:05:06.423875 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:06.423837 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:05:06.522926 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:06.522893 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:05:16.241388 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:16.241338 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:05:20.852044 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:20.852002 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:05:21.294072 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:21.293969 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:05:23.220399 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:23.220365 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:05:23.321384 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:23.321342 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:05:23.540843 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:23.540743 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:05:27.009183 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:27.009142 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:05:27.109878 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:27.109841 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:05:27.248714 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:27.248676 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9"] Jul 21 01:05:27.248979 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:27.248940 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" podUID="debbff81-ed54-4548-b64d-737eeaffd528" containerName="istio-proxy" containerID="cri-o://821331ac6cd07971ad16f933831919a5d43a683dde9b71e55c1db5e257303c6e" gracePeriod=30 Jul 21 01:05:27.270223 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:27.270135 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:05:27.523395 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:27.523308 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:05:28.190565 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:28.190532 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:05:28.293137 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:28.293095 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:05:32.378985 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.378949 2580 generic.go:356] "Generic (PLEG): container finished" podID="debbff81-ed54-4548-b64d-737eeaffd528" containerID="821331ac6cd07971ad16f933831919a5d43a683dde9b71e55c1db5e257303c6e" exitCode=0 Jul 21 01:05:32.379453 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.379018 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" event={"ID":"debbff81-ed54-4548-b64d-737eeaffd528","Type":"ContainerDied","Data":"821331ac6cd07971ad16f933831919a5d43a683dde9b71e55c1db5e257303c6e"} Jul 21 01:05:32.379453 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.379083 2580 scope.go:117] "RemoveContainer" containerID="b9dca32ccf29821d557e480369a2424314e9aed1542da84aec61687e542836e8" Jul 21 01:05:32.518257 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.518161 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 01:05:32.596680 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.596639 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wh4d\" (UniqueName: \"kubernetes.io/projected/debbff81-ed54-4548-b64d-737eeaffd528-kube-api-access-2wh4d\") pod \"debbff81-ed54-4548-b64d-737eeaffd528\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " Jul 21 01:05:32.596900 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.596729 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-workload-certs\") pod \"debbff81-ed54-4548-b64d-737eeaffd528\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " Jul 21 01:05:32.596900 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.596787 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-istio-envoy\") pod \"debbff81-ed54-4548-b64d-737eeaffd528\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " Jul 21 01:05:32.596900 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.596811 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-istio-data\") pod \"debbff81-ed54-4548-b64d-737eeaffd528\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " Jul 21 01:05:32.596900 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.596850 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debbff81-ed54-4548-b64d-737eeaffd528-istio-podinfo\") pod \"debbff81-ed54-4548-b64d-737eeaffd528\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " Jul 21 01:05:32.596900 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.596873 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debbff81-ed54-4548-b64d-737eeaffd528-istiod-ca-cert\") pod \"debbff81-ed54-4548-b64d-737eeaffd528\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " Jul 21 01:05:32.597183 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.596906 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-workload-socket\") pod \"debbff81-ed54-4548-b64d-737eeaffd528\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " Jul 21 01:05:32.597183 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.596934 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/debbff81-ed54-4548-b64d-737eeaffd528-istio-token\") pod \"debbff81-ed54-4548-b64d-737eeaffd528\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " Jul 21 01:05:32.597183 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.596967 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-credential-socket\") pod \"debbff81-ed54-4548-b64d-737eeaffd528\" (UID: \"debbff81-ed54-4548-b64d-737eeaffd528\") " Jul 21 01:05:32.597183 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.597006 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "debbff81-ed54-4548-b64d-737eeaffd528" (UID: "debbff81-ed54-4548-b64d-737eeaffd528"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:05:32.597409 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.597304 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-workload-certs\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:05:32.597471 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.597406 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/debbff81-ed54-4548-b64d-737eeaffd528-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "debbff81-ed54-4548-b64d-737eeaffd528" (UID: "debbff81-ed54-4548-b64d-737eeaffd528"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 01:05:32.597743 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.597664 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "debbff81-ed54-4548-b64d-737eeaffd528" (UID: "debbff81-ed54-4548-b64d-737eeaffd528"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:05:32.597743 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.597666 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "debbff81-ed54-4548-b64d-737eeaffd528" (UID: "debbff81-ed54-4548-b64d-737eeaffd528"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:05:32.597973 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.597823 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-istio-data" (OuterVolumeSpecName: "istio-data") pod "debbff81-ed54-4548-b64d-737eeaffd528" (UID: "debbff81-ed54-4548-b64d-737eeaffd528"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:05:32.599118 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.599089 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/debbff81-ed54-4548-b64d-737eeaffd528-kube-api-access-2wh4d" (OuterVolumeSpecName: "kube-api-access-2wh4d") pod "debbff81-ed54-4548-b64d-737eeaffd528" (UID: "debbff81-ed54-4548-b64d-737eeaffd528"). InnerVolumeSpecName "kube-api-access-2wh4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:05:32.599233 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.599164 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "debbff81-ed54-4548-b64d-737eeaffd528" (UID: "debbff81-ed54-4548-b64d-737eeaffd528"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:05:32.599278 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.599235 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/debbff81-ed54-4548-b64d-737eeaffd528-istio-token" (OuterVolumeSpecName: "istio-token") pod "debbff81-ed54-4548-b64d-737eeaffd528" (UID: "debbff81-ed54-4548-b64d-737eeaffd528"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:05:32.599368 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.599348 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/debbff81-ed54-4548-b64d-737eeaffd528-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "debbff81-ed54-4548-b64d-737eeaffd528" (UID: "debbff81-ed54-4548-b64d-737eeaffd528"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 01:05:32.698724 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.698667 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wh4d\" (UniqueName: \"kubernetes.io/projected/debbff81-ed54-4548-b64d-737eeaffd528-kube-api-access-2wh4d\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:05:32.698724 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.698716 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-istio-envoy\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:05:32.698724 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.698727 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-istio-data\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:05:32.698724 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.698735 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/debbff81-ed54-4548-b64d-737eeaffd528-istio-podinfo\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:05:32.698724 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.698743 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/debbff81-ed54-4548-b64d-737eeaffd528-istiod-ca-cert\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:05:32.699049 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.698752 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-workload-socket\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:05:32.699049 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.698760 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/debbff81-ed54-4548-b64d-737eeaffd528-istio-token\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:05:32.699049 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:32.698768 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/debbff81-ed54-4548-b64d-737eeaffd528-credential-socket\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:05:33.387089 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:33.387060 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" Jul 21 01:05:33.387089 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:33.387071 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9" event={"ID":"debbff81-ed54-4548-b64d-737eeaffd528","Type":"ContainerDied","Data":"016fb001a8095eccc5d165df9eb4e634af3b855a50668feca3313aa5015cb320"} Jul 21 01:05:33.387626 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:33.387136 2580 scope.go:117] "RemoveContainer" containerID="821331ac6cd07971ad16f933831919a5d43a683dde9b71e55c1db5e257303c6e" Jul 21 01:05:33.410717 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:33.410677 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9"] Jul 21 01:05:33.415306 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:33.415270 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-16b734-openshift-default-67cc67757f-nntx9"] Jul 21 01:05:34.298414 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:34.298375 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="debbff81-ed54-4548-b64d-737eeaffd528" path="/var/lib/kubelet/pods/debbff81-ed54-4548-b64d-737eeaffd528/volumes" Jul 21 01:05:38.301073 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:38.301042 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:05:42.733441 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:42.733401 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:05:43.159368 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:43.159324 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:05:49.247312 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:49.247265 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:05:49.288783 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:49.288740 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g"] Jul 21 01:05:49.289041 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:49.289008 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" podUID="0c5b99bc-61b0-4917-9df1-c52583bb008d" containerName="istio-proxy" containerID="cri-o://d782b8ea22758dbbf3e0975dd83d4c4fde9c83d2d04acdbf562a03e3ffea62f4" gracePeriod=30 Jul 21 01:05:49.312624 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:49.312570 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:05:49.551718 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:49.551630 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:05:54.515125 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.515082 2580 generic.go:356] "Generic (PLEG): container finished" podID="0c5b99bc-61b0-4917-9df1-c52583bb008d" containerID="d782b8ea22758dbbf3e0975dd83d4c4fde9c83d2d04acdbf562a03e3ffea62f4" exitCode=0 Jul 21 01:05:54.515555 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.515157 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" event={"ID":"0c5b99bc-61b0-4917-9df1-c52583bb008d","Type":"ContainerDied","Data":"d782b8ea22758dbbf3e0975dd83d4c4fde9c83d2d04acdbf562a03e3ffea62f4"} Jul 21 01:05:54.515555 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.515225 2580 scope.go:117] "RemoveContainer" containerID="bc595f3d7489364df6ef6ea7af682e86050d11c2ca3674efe4d9f6bc825a9845" Jul 21 01:05:54.563908 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.563836 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 01:05:54.714515 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.714473 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c5b99bc-61b0-4917-9df1-c52583bb008d-istiod-ca-cert\") pod \"0c5b99bc-61b0-4917-9df1-c52583bb008d\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " Jul 21 01:05:54.714751 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.714535 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-credential-socket\") pod \"0c5b99bc-61b0-4917-9df1-c52583bb008d\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " Jul 21 01:05:54.714751 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.714573 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr74q\" (UniqueName: \"kubernetes.io/projected/0c5b99bc-61b0-4917-9df1-c52583bb008d-kube-api-access-kr74q\") pod \"0c5b99bc-61b0-4917-9df1-c52583bb008d\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " Jul 21 01:05:54.714751 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.714634 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-istio-data\") pod \"0c5b99bc-61b0-4917-9df1-c52583bb008d\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " Jul 21 01:05:54.714751 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.714656 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-workload-certs\") pod \"0c5b99bc-61b0-4917-9df1-c52583bb008d\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " Jul 21 01:05:54.714751 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.714678 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c5b99bc-61b0-4917-9df1-c52583bb008d-istio-podinfo\") pod \"0c5b99bc-61b0-4917-9df1-c52583bb008d\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " Jul 21 01:05:54.714751 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.714702 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-istio-envoy\") pod \"0c5b99bc-61b0-4917-9df1-c52583bb008d\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " Jul 21 01:05:54.714751 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.714738 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c5b99bc-61b0-4917-9df1-c52583bb008d-istio-token\") pod \"0c5b99bc-61b0-4917-9df1-c52583bb008d\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " Jul 21 01:05:54.715403 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.714798 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-workload-socket\") pod \"0c5b99bc-61b0-4917-9df1-c52583bb008d\" (UID: \"0c5b99bc-61b0-4917-9df1-c52583bb008d\") " Jul 21 01:05:54.715403 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.714931 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0c5b99bc-61b0-4917-9df1-c52583bb008d" (UID: "0c5b99bc-61b0-4917-9df1-c52583bb008d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:05:54.715403 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.714981 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0c5b99bc-61b0-4917-9df1-c52583bb008d" (UID: "0c5b99bc-61b0-4917-9df1-c52583bb008d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:05:54.715403 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.715053 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c5b99bc-61b0-4917-9df1-c52583bb008d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0c5b99bc-61b0-4917-9df1-c52583bb008d" (UID: "0c5b99bc-61b0-4917-9df1-c52583bb008d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 01:05:54.715403 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.715157 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0c5b99bc-61b0-4917-9df1-c52583bb008d" (UID: "0c5b99bc-61b0-4917-9df1-c52583bb008d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:05:54.715403 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.715227 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c5b99bc-61b0-4917-9df1-c52583bb008d-istiod-ca-cert\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:05:54.715403 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.715245 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-credential-socket\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:05:54.715403 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.715256 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-workload-certs\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:05:54.715933 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.715906 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-istio-data" (OuterVolumeSpecName: "istio-data") pod "0c5b99bc-61b0-4917-9df1-c52583bb008d" (UID: "0c5b99bc-61b0-4917-9df1-c52583bb008d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:05:54.717102 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.717067 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c5b99bc-61b0-4917-9df1-c52583bb008d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0c5b99bc-61b0-4917-9df1-c52583bb008d" (UID: "0c5b99bc-61b0-4917-9df1-c52583bb008d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 01:05:54.717102 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.717097 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c5b99bc-61b0-4917-9df1-c52583bb008d-istio-token" (OuterVolumeSpecName: "istio-token") pod "0c5b99bc-61b0-4917-9df1-c52583bb008d" (UID: "0c5b99bc-61b0-4917-9df1-c52583bb008d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:05:54.717505 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.717477 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0c5b99bc-61b0-4917-9df1-c52583bb008d" (UID: "0c5b99bc-61b0-4917-9df1-c52583bb008d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 01:05:54.717505 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.717489 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c5b99bc-61b0-4917-9df1-c52583bb008d-kube-api-access-kr74q" (OuterVolumeSpecName: "kube-api-access-kr74q") pod "0c5b99bc-61b0-4917-9df1-c52583bb008d" (UID: "0c5b99bc-61b0-4917-9df1-c52583bb008d"). InnerVolumeSpecName "kube-api-access-kr74q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 01:05:54.816793 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.816703 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c5b99bc-61b0-4917-9df1-c52583bb008d-istio-token\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:05:54.816793 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.816734 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-workload-socket\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:05:54.816793 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.816743 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr74q\" (UniqueName: \"kubernetes.io/projected/0c5b99bc-61b0-4917-9df1-c52583bb008d-kube-api-access-kr74q\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:05:54.816793 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.816753 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-istio-data\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:05:54.816793 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.816762 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c5b99bc-61b0-4917-9df1-c52583bb008d-istio-podinfo\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:05:54.816793 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:54.816770 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c5b99bc-61b0-4917-9df1-c52583bb008d-istio-envoy\") on node \"ip-10-0-139-117.ec2.internal\" DevicePath \"\"" Jul 21 01:05:55.522761 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:55.522730 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" Jul 21 01:05:55.523236 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:55.522768 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g" event={"ID":"0c5b99bc-61b0-4917-9df1-c52583bb008d","Type":"ContainerDied","Data":"73e6abe6fc0279aa553ef2b2ca8e925aeec89e67d9b015c961362133971230d3"} Jul 21 01:05:55.523236 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:55.522832 2580 scope.go:117] "RemoveContainer" containerID="d782b8ea22758dbbf3e0975dd83d4c4fde9c83d2d04acdbf562a03e3ffea62f4" Jul 21 01:05:55.544733 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:55.544687 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g"] Jul 21 01:05:55.549236 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:55.549196 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-b4ed4b-openshift-default-7cc7594d6c-r2d2g"] Jul 21 01:05:55.734520 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:55.734494 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xpb5v_85364ca8-8d29-4a95-a39c-b9ea72ab6873/manager/0.log" Jul 21 01:05:55.862571 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:55.862534 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-s6fs8_01b985e5-79f1-4822-b3d5-93bf24f5599e/manager/0.log" Jul 21 01:05:55.984272 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:55.984240 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-79bfd5d9f6-d9ft9_d75f19e5-a207-4df5-a72e-9782ba3927a8/manager/0.log" Jul 21 01:05:56.103858 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:56.103831 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-2tln2_6a39002d-d7f1-4023-a125-2832e32db4f7/server/0.log" Jul 21 01:05:56.228962 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:56.228868 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-h24w6_8ce97c9b-8333-4464-a317-33fec0ef2813/manager/0.log" Jul 21 01:05:56.297957 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:56.297920 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c5b99bc-61b0-4917-9df1-c52583bb008d" path="/var/lib/kubelet/pods/0c5b99bc-61b0-4917-9df1-c52583bb008d/volumes" Jul 21 01:05:56.346505 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:56.346478 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7_49be6af6-766d-431d-b0cd-0963a5695e30/copy-manifests/0.log" Jul 21 01:05:56.352873 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:56.352841 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7_49be6af6-766d-431d-b0cd-0963a5695e30/manager/0.log" Jul 21 01:05:57.002944 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:57.002910 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-658c477b49-fw2r6_fa980c65-1c48-4872-bae1-082a7aeda718/maas-api/0.log" Jul 21 01:05:57.131381 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:57.131351 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29743230-wddzw_f1bab934-69c0-4bfd-abe3-4961ff78b70a/cleanup/0.log" Jul 21 01:05:57.249190 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:57.249163 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29743245-sjvv4_fbd4ac73-6cd5-4f40-964c-c584155856c8/cleanup/0.log" Jul 21 01:05:57.480426 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:57.480390 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-dw8m6_12b4b9f5-5148-400d-834a-f1822f8eacfb/postgres/0.log" Jul 21 01:05:58.310778 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:58.310742 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms_b6a041bf-6dae-4b00-9fb7-30b07b16bdbd/util/0.log" Jul 21 01:05:58.317659 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:58.317622 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms_b6a041bf-6dae-4b00-9fb7-30b07b16bdbd/pull/0.log" Jul 21 01:05:58.324900 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:58.324877 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms_b6a041bf-6dae-4b00-9fb7-30b07b16bdbd/extract/0.log" Jul 21 01:05:58.447193 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:58.447166 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl_dd296917-fa8f-471e-b4fc-9795c3be221d/util/0.log" Jul 21 01:05:58.453836 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:58.453810 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl_dd296917-fa8f-471e-b4fc-9795c3be221d/pull/0.log" Jul 21 01:05:58.460922 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:58.460899 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl_dd296917-fa8f-471e-b4fc-9795c3be221d/extract/0.log" Jul 21 01:05:58.582390 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:58.582307 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn_d48e193a-a228-4bfa-8bb0-b756f456bf10/util/0.log" Jul 21 01:05:58.588830 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:58.588807 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn_d48e193a-a228-4bfa-8bb0-b756f456bf10/pull/0.log" Jul 21 01:05:58.595084 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:58.595064 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn_d48e193a-a228-4bfa-8bb0-b756f456bf10/extract/0.log" Jul 21 01:05:58.710186 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:58.710151 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7_1f66e031-60a1-4f6d-942b-782770b2ea3d/extract/0.log" Jul 21 01:05:58.716663 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:58.716637 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7_1f66e031-60a1-4f6d-942b-782770b2ea3d/util/0.log" Jul 21 01:05:58.722969 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:58.722946 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7_1f66e031-60a1-4f6d-942b-782770b2ea3d/pull/0.log" Jul 21 01:05:58.969224 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:58.969138 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-s8c42_43e79b6a-729d-4a56-96c1-10d0e6e6f935/manager/0.log" Jul 21 01:05:59.089061 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:59.089031 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-27jbj_1705926d-8894-488d-add2-121845ea0ff7/manager/0.log" Jul 21 01:05:59.328762 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:59.328726 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-zfgtr_477c7451-09be-447d-930b-3c005d6660c7/registry-server/0.log" Jul 21 01:05:59.687363 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:05:59.687272 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-rsfpd_0fa997ae-55a5-49ac-833d-88e3c070f32c/manager/0.log" Jul 21 01:06:00.054977 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:00.054944 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4nzxzx_4d01be36-93d9-4180-90cf-ca8f7fde13b9/istio-proxy/0.log" Jul 21 01:06:00.290111 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:00.290081 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66766cb76c-plvdj_bf25e5c5-65e1-472c-b2d0-0623a79a7e4e/kube-auth-proxy/0.log" Jul 21 01:06:00.534180 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:00.534090 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-6vs2d_944990f8-eece-4bbe-a092-1db9b9547c2f/istio-proxy/1.log" Jul 21 01:06:00.790708 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:00.790621 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-lxrfg_9f3e5dde-da98-4b74-aed1-d9351b3e325c/payload-processing/0.log" Jul 21 01:06:01.416104 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:01.416077 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-mzvhl_51fa7b53-6a12-405e-aa14-30e3c86d7869/main/0.log" Jul 21 01:06:01.530782 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:01.530751 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-lgvkn_4290eeee-cec8-49d5-b4cc-a02807ed098f/main/0.log" Jul 21 01:06:01.885048 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:01.885021 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-f4ms7_bc686d9a-c5b8-4792-ac40-99ac7151ad49/main/0.log" Jul 21 01:06:07.587710 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.587580 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sf6l8/must-gather-ccv6f"] Jul 21 01:06:07.588166 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.588072 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c5b99bc-61b0-4917-9df1-c52583bb008d" containerName="istio-proxy" Jul 21 01:06:07.588166 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.588087 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c5b99bc-61b0-4917-9df1-c52583bb008d" containerName="istio-proxy" Jul 21 01:06:07.588166 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.588097 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="554e14d0-bd94-481e-b7cf-c2c8a3872d01" containerName="test-curl-7191-140002104607056" Jul 21 01:06:07.588166 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.588107 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="554e14d0-bd94-481e-b7cf-c2c8a3872d01" containerName="test-curl-7191-140002104607056" Jul 21 01:06:07.588166 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.588117 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="debbff81-ed54-4548-b64d-737eeaffd528" containerName="istio-proxy" Jul 21 01:06:07.588166 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.588123 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="debbff81-ed54-4548-b64d-737eeaffd528" containerName="istio-proxy" Jul 21 01:06:07.588166 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.588145 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="debbff81-ed54-4548-b64d-737eeaffd528" containerName="istio-proxy" Jul 21 01:06:07.588166 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.588150 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="debbff81-ed54-4548-b64d-737eeaffd528" containerName="istio-proxy" Jul 21 01:06:07.588592 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.588234 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c5b99bc-61b0-4917-9df1-c52583bb008d" containerName="istio-proxy" Jul 21 01:06:07.588592 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.588244 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c5b99bc-61b0-4917-9df1-c52583bb008d" containerName="istio-proxy" Jul 21 01:06:07.588592 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.588251 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="debbff81-ed54-4548-b64d-737eeaffd528" containerName="istio-proxy" Jul 21 01:06:07.588592 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.588258 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="554e14d0-bd94-481e-b7cf-c2c8a3872d01" containerName="test-curl-7191-140002104607056" Jul 21 01:06:07.588592 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.588270 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="debbff81-ed54-4548-b64d-737eeaffd528" containerName="istio-proxy" Jul 21 01:06:07.588592 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.588383 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c5b99bc-61b0-4917-9df1-c52583bb008d" containerName="istio-proxy" Jul 21 01:06:07.588592 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.588395 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c5b99bc-61b0-4917-9df1-c52583bb008d" containerName="istio-proxy" Jul 21 01:06:07.593150 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.593126 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sf6l8/must-gather-ccv6f" Jul 21 01:06:07.597938 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.597913 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sf6l8\"/\"kube-root-ca.crt\"" Jul 21 01:06:07.598195 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.598172 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sf6l8\"/\"default-dockercfg-pslfp\"" Jul 21 01:06:07.598887 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.598867 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sf6l8\"/\"openshift-service-ca.crt\"" Jul 21 01:06:07.602334 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.602310 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sf6l8/must-gather-ccv6f"] Jul 21 01:06:07.608847 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.608819 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:06:07.639769 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.639722 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpsrk\" (UniqueName: \"kubernetes.io/projected/a96f21ac-f01e-49e5-bdc1-4c37d5b9eca2-kube-api-access-fpsrk\") pod \"must-gather-ccv6f\" (UID: \"a96f21ac-f01e-49e5-bdc1-4c37d5b9eca2\") " pod="openshift-must-gather-sf6l8/must-gather-ccv6f" Jul 21 01:06:07.639769 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.639777 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a96f21ac-f01e-49e5-bdc1-4c37d5b9eca2-must-gather-output\") pod \"must-gather-ccv6f\" (UID: \"a96f21ac-f01e-49e5-bdc1-4c37d5b9eca2\") " pod="openshift-must-gather-sf6l8/must-gather-ccv6f" Jul 21 01:06:07.740689 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.740641 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a96f21ac-f01e-49e5-bdc1-4c37d5b9eca2-must-gather-output\") pod \"must-gather-ccv6f\" (UID: \"a96f21ac-f01e-49e5-bdc1-4c37d5b9eca2\") " pod="openshift-must-gather-sf6l8/must-gather-ccv6f" Jul 21 01:06:07.740874 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.740764 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpsrk\" (UniqueName: \"kubernetes.io/projected/a96f21ac-f01e-49e5-bdc1-4c37d5b9eca2-kube-api-access-fpsrk\") pod \"must-gather-ccv6f\" (UID: \"a96f21ac-f01e-49e5-bdc1-4c37d5b9eca2\") " pod="openshift-must-gather-sf6l8/must-gather-ccv6f" Jul 21 01:06:07.740988 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.740966 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a96f21ac-f01e-49e5-bdc1-4c37d5b9eca2-must-gather-output\") pod \"must-gather-ccv6f\" (UID: \"a96f21ac-f01e-49e5-bdc1-4c37d5b9eca2\") " pod="openshift-must-gather-sf6l8/must-gather-ccv6f" Jul 21 01:06:07.749762 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.749737 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpsrk\" (UniqueName: \"kubernetes.io/projected/a96f21ac-f01e-49e5-bdc1-4c37d5b9eca2-kube-api-access-fpsrk\") pod \"must-gather-ccv6f\" (UID: \"a96f21ac-f01e-49e5-bdc1-4c37d5b9eca2\") " pod="openshift-must-gather-sf6l8/must-gather-ccv6f" Jul 21 01:06:07.904049 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:07.903951 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sf6l8/must-gather-ccv6f" Jul 21 01:06:08.049512 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:08.049490 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sf6l8/must-gather-ccv6f"] Jul 21 01:06:08.051997 ip-10-0-139-117 kubenswrapper[2580]: W0721 01:06:08.051968 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda96f21ac_f01e_49e5_bdc1_4c37d5b9eca2.slice/crio-84cd8bfd2aa1c88444cc6d943fec96be8edb57afcd9aec4cdb0b4c3f1c5d2a31 WatchSource:0}: Error finding container 84cd8bfd2aa1c88444cc6d943fec96be8edb57afcd9aec4cdb0b4c3f1c5d2a31: Status 404 returned error can't find the container with id 84cd8bfd2aa1c88444cc6d943fec96be8edb57afcd9aec4cdb0b4c3f1c5d2a31 Jul 21 01:06:08.287210 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:08.287124 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:06:08.600778 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:08.600747 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sf6l8/must-gather-ccv6f" event={"ID":"a96f21ac-f01e-49e5-bdc1-4c37d5b9eca2","Type":"ContainerStarted","Data":"84cd8bfd2aa1c88444cc6d943fec96be8edb57afcd9aec4cdb0b4c3f1c5d2a31"} Jul 21 01:06:09.611517 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:09.611477 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sf6l8/must-gather-ccv6f" event={"ID":"a96f21ac-f01e-49e5-bdc1-4c37d5b9eca2","Type":"ContainerStarted","Data":"4aefd6a5db1bf3362c9879ba4c4521d9c41920459dd4c70d73f136afe9e633ca"} Jul 21 01:06:09.612056 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:09.611533 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sf6l8/must-gather-ccv6f" event={"ID":"a96f21ac-f01e-49e5-bdc1-4c37d5b9eca2","Type":"ContainerStarted","Data":"f3fbd8294a591aaab0fe48f42ce13fc008dec2eab0b4d38263cc81cbc6a0448a"} Jul 21 01:06:09.632362 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:09.632283 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sf6l8/must-gather-ccv6f" podStartSLOduration=1.666112448 podStartE2EDuration="2.632256732s" totalImagesPullingTime="966.144284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 01:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:06:09.627928404 +0000 UTC m=+2963.843308730" watchObservedRunningTime="2026-07-21 01:06:09.632256732 +0000 UTC m=+2963.847637057" Jul 21 01:06:10.578853 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:10.578825 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2psfz_23335158-224d-46e1-b617-10ffa4acb460/global-pull-secret-syncer/0.log" Jul 21 01:06:10.719612 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:10.719575 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" Jul 21 01:06:10.804300 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:10.804271 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-117.ec2.internal_30cbc68e3783a788ccf4c7e2f8cc9298/haproxy/0.log" Jul 21 01:06:15.177934 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:15.177895 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms_b6a041bf-6dae-4b00-9fb7-30b07b16bdbd/extract/0.log" Jul 21 01:06:15.204958 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:15.204912 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms_b6a041bf-6dae-4b00-9fb7-30b07b16bdbd/util/0.log" Jul 21 01:06:15.232665 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:15.232632 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759z4lms_b6a041bf-6dae-4b00-9fb7-30b07b16bdbd/pull/0.log" Jul 21 01:06:15.263349 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:15.263318 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl_dd296917-fa8f-471e-b4fc-9795c3be221d/extract/0.log" Jul 21 01:06:15.287412 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:15.287383 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl_dd296917-fa8f-471e-b4fc-9795c3be221d/util/0.log" Jul 21 01:06:15.318919 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:15.318888 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nlndl_dd296917-fa8f-471e-b4fc-9795c3be221d/pull/0.log" Jul 21 01:06:15.365897 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:15.365837 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn_d48e193a-a228-4bfa-8bb0-b756f456bf10/extract/0.log" Jul 21 01:06:15.396160 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:15.396137 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn_d48e193a-a228-4bfa-8bb0-b756f456bf10/util/0.log" Jul 21 01:06:15.421523 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:15.421377 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73cswcn_d48e193a-a228-4bfa-8bb0-b756f456bf10/pull/0.log" Jul 21 01:06:15.451696 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:15.451513 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7_1f66e031-60a1-4f6d-942b-782770b2ea3d/extract/0.log" Jul 21 01:06:15.474425 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:15.474400 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7_1f66e031-60a1-4f6d-942b-782770b2ea3d/util/0.log" Jul 21 01:06:15.502380 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:15.502349 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lhvl7_1f66e031-60a1-4f6d-942b-782770b2ea3d/pull/0.log" Jul 21 01:06:15.868119 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:15.868082 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-s8c42_43e79b6a-729d-4a56-96c1-10d0e6e6f935/manager/0.log" Jul 21 01:06:15.910447 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:15.910392 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-27jbj_1705926d-8894-488d-add2-121845ea0ff7/manager/0.log" Jul 21 01:06:15.983552 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:15.983510 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-zfgtr_477c7451-09be-447d-930b-3c005d6660c7/registry-server/0.log" Jul 21 01:06:16.223236 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:16.223131 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-rsfpd_0fa997ae-55a5-49ac-833d-88e3c070f32c/manager/0.log" Jul 21 01:06:18.072176 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:18.072040 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jlx72_da6931b3-d9b0-4c63-a0ba-7eca03b046db/konnectivity-agent/0.log" limit=1024 Jul 21 01:06:18.162232 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:18.162090 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-ccs8p_13b254c8-1a1e-40db-b169-9c13a98430f5/monitoring-plugin/0.log" Jul 21 01:06:18.299710 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:18.299677 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f9hxl_bb7d385b-b316-481f-9cad-354c798728cc/node-exporter/0.log" Jul 21 01:06:18.327396 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:18.327307 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f9hxl_bb7d385b-b316-481f-9cad-354c798728cc/kube-rbac-proxy/0.log" Jul 21 01:06:18.605997 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:18.605891 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f9hxl_bb7d385b-b316-481f-9cad-354c798728cc/init-textfile/0.log" Jul 21 01:06:19.036714 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:19.036584 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-j7bpq_2beddce2-7b22-48ef-9399-df9c9c0f9be4/prometheus-operator-admission-webhook/0.log" Jul 21 01:06:19.148773 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:19.148740 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59b7fcc89f-frkvp_d9783a8f-d1a0-4d04-8452-e8cc6520cf5e/thanos-query/0.log" Jul 21 01:06:19.174755 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:19.174699 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59b7fcc89f-frkvp_d9783a8f-d1a0-4d04-8452-e8cc6520cf5e/kube-rbac-proxy-web/0.log" Jul 21 01:06:19.200935 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:19.200908 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59b7fcc89f-frkvp_d9783a8f-d1a0-4d04-8452-e8cc6520cf5e/kube-rbac-proxy/0.log" Jul 21 01:06:19.223563 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:19.223527 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59b7fcc89f-frkvp_d9783a8f-d1a0-4d04-8452-e8cc6520cf5e/prom-label-proxy/0.log" Jul 21 01:06:19.252855 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:19.252820 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59b7fcc89f-frkvp_d9783a8f-d1a0-4d04-8452-e8cc6520cf5e/kube-rbac-proxy-rules/0.log" Jul 21 01:06:19.275942 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:19.275911 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59b7fcc89f-frkvp_d9783a8f-d1a0-4d04-8452-e8cc6520cf5e/kube-rbac-proxy-metrics/0.log" Jul 21 01:06:21.021762 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.021727 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb"] Jul 21 01:06:21.031494 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.031460 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb" Jul 21 01:06:21.038043 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.038007 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb"] Jul 21 01:06:21.075138 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.075085 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cce448e4-752a-44bf-b6ce-fe5def9c96a8-podres\") pod \"perf-node-gather-daemonset-rcqcb\" (UID: \"cce448e4-752a-44bf-b6ce-fe5def9c96a8\") " pod="openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb" Jul 21 01:06:21.075341 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.075219 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cce448e4-752a-44bf-b6ce-fe5def9c96a8-sys\") pod \"perf-node-gather-daemonset-rcqcb\" (UID: \"cce448e4-752a-44bf-b6ce-fe5def9c96a8\") " pod="openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb" Jul 21 01:06:21.075341 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.075255 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mtg8\" (UniqueName: \"kubernetes.io/projected/cce448e4-752a-44bf-b6ce-fe5def9c96a8-kube-api-access-5mtg8\") pod \"perf-node-gather-daemonset-rcqcb\" (UID: \"cce448e4-752a-44bf-b6ce-fe5def9c96a8\") " pod="openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb" Jul 21 01:06:21.075341 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.075282 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cce448e4-752a-44bf-b6ce-fe5def9c96a8-lib-modules\") pod \"perf-node-gather-daemonset-rcqcb\" (UID: \"cce448e4-752a-44bf-b6ce-fe5def9c96a8\") " pod="openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb" Jul 21 01:06:21.075526 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.075376 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cce448e4-752a-44bf-b6ce-fe5def9c96a8-proc\") pod \"perf-node-gather-daemonset-rcqcb\" (UID: \"cce448e4-752a-44bf-b6ce-fe5def9c96a8\") " pod="openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb" Jul 21 01:06:21.176157 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.176113 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cce448e4-752a-44bf-b6ce-fe5def9c96a8-sys\") pod \"perf-node-gather-daemonset-rcqcb\" (UID: \"cce448e4-752a-44bf-b6ce-fe5def9c96a8\") " pod="openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb" Jul 21 01:06:21.176462 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.176420 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mtg8\" (UniqueName: \"kubernetes.io/projected/cce448e4-752a-44bf-b6ce-fe5def9c96a8-kube-api-access-5mtg8\") pod \"perf-node-gather-daemonset-rcqcb\" (UID: \"cce448e4-752a-44bf-b6ce-fe5def9c96a8\") " pod="openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb" Jul 21 01:06:21.176690 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.176670 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cce448e4-752a-44bf-b6ce-fe5def9c96a8-lib-modules\") pod \"perf-node-gather-daemonset-rcqcb\" (UID: \"cce448e4-752a-44bf-b6ce-fe5def9c96a8\") " pod="openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb" Jul 21 01:06:21.176943 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.176925 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cce448e4-752a-44bf-b6ce-fe5def9c96a8-proc\") pod \"perf-node-gather-daemonset-rcqcb\" (UID: \"cce448e4-752a-44bf-b6ce-fe5def9c96a8\") " pod="openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb" Jul 21 01:06:21.177103 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.177087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cce448e4-752a-44bf-b6ce-fe5def9c96a8-podres\") pod \"perf-node-gather-daemonset-rcqcb\" (UID: \"cce448e4-752a-44bf-b6ce-fe5def9c96a8\") " pod="openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb" Jul 21 01:06:21.177672 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.177534 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cce448e4-752a-44bf-b6ce-fe5def9c96a8-podres\") pod \"perf-node-gather-daemonset-rcqcb\" (UID: \"cce448e4-752a-44bf-b6ce-fe5def9c96a8\") " pod="openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb" Jul 21 01:06:21.177672 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.177532 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cce448e4-752a-44bf-b6ce-fe5def9c96a8-lib-modules\") pod \"perf-node-gather-daemonset-rcqcb\" (UID: \"cce448e4-752a-44bf-b6ce-fe5def9c96a8\") " pod="openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb" Jul 21 01:06:21.177672 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.177644 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cce448e4-752a-44bf-b6ce-fe5def9c96a8-sys\") pod \"perf-node-gather-daemonset-rcqcb\" (UID: \"cce448e4-752a-44bf-b6ce-fe5def9c96a8\") " pod="openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb" Jul 21 01:06:21.177672 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.177668 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cce448e4-752a-44bf-b6ce-fe5def9c96a8-proc\") pod \"perf-node-gather-daemonset-rcqcb\" (UID: \"cce448e4-752a-44bf-b6ce-fe5def9c96a8\") " pod="openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb" Jul 21 01:06:21.191635 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.189004 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mtg8\" (UniqueName: \"kubernetes.io/projected/cce448e4-752a-44bf-b6ce-fe5def9c96a8-kube-api-access-5mtg8\") pod \"perf-node-gather-daemonset-rcqcb\" (UID: \"cce448e4-752a-44bf-b6ce-fe5def9c96a8\") " pod="openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb" Jul 21 01:06:21.346430 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.346385 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb" Jul 21 01:06:21.561824 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.561762 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb"] Jul 21 01:06:21.563531 ip-10-0-139-117 kubenswrapper[2580]: W0721 01:06:21.563470 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcce448e4_752a_44bf_b6ce_fe5def9c96a8.slice/crio-74ced08b67b00b1e659c8b7f461a6de5cfd2b51d50d5de5c1739497e6f0d15d0 WatchSource:0}: Error finding container 74ced08b67b00b1e659c8b7f461a6de5cfd2b51d50d5de5c1739497e6f0d15d0: Status 404 returned error can't find the container with id 74ced08b67b00b1e659c8b7f461a6de5cfd2b51d50d5de5c1739497e6f0d15d0 Jul 21 01:06:21.705216 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.705190 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5944955d98-bhzfc_9351e4d7-fd14-498b-b85f-8a714dcf8df6/console/0.log" Jul 21 01:06:21.716823 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.716786 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb" event={"ID":"cce448e4-752a-44bf-b6ce-fe5def9c96a8","Type":"ContainerStarted","Data":"261a1425b2ac4a1468d609e3e3ace604b6dbe9edf0113d023f8c9de9b447212f"} Jul 21 01:06:21.717043 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.716850 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb" event={"ID":"cce448e4-752a-44bf-b6ce-fe5def9c96a8","Type":"ContainerStarted","Data":"74ced08b67b00b1e659c8b7f461a6de5cfd2b51d50d5de5c1739497e6f0d15d0"} Jul 21 01:06:21.717043 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.717008 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb" Jul 21 01:06:21.736753 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:21.736693 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb" podStartSLOduration=0.736675824 podStartE2EDuration="736.675824ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 01:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 01:06:21.734523944 +0000 UTC m=+2975.949904270" watchObservedRunningTime="2026-07-21 01:06:21.736675824 +0000 UTC m=+2975.952056149" Jul 21 01:06:23.461263 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:23.461233 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pz4d4_59620b96-9827-4757-841e-be5c630ef424/dns/0.log" Jul 21 01:06:23.483942 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:23.483892 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pz4d4_59620b96-9827-4757-841e-be5c630ef424/kube-rbac-proxy/0.log" Jul 21 01:06:23.613154 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:23.613129 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-szxbf_583c4be0-d3e2-4a43-b91c-d6a8903569da/dns-node-resolver/0.log" Jul 21 01:06:24.214270 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:24.214232 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p76tl_2bd25715-f83d-42fa-953e-ba07d8b7da50/node-ca/0.log" Jul 21 01:06:25.175448 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:25.175408 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4nzxzx_4d01be36-93d9-4180-90cf-ca8f7fde13b9/istio-proxy/0.log" Jul 21 01:06:25.330186 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:25.330158 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66766cb76c-plvdj_bf25e5c5-65e1-472c-b2d0-0623a79a7e4e/kube-auth-proxy/0.log" Jul 21 01:06:25.404620 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:25.404567 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-6vs2d_944990f8-eece-4bbe-a092-1db9b9547c2f/istio-proxy/0.log" Jul 21 01:06:25.442136 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:25.442014 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-6vs2d_944990f8-eece-4bbe-a092-1db9b9547c2f/istio-proxy/1.log" Jul 21 01:06:25.504945 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:25.504886 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-lxrfg_9f3e5dde-da98-4b74-aed1-d9351b3e325c/payload-processing/0.log" Jul 21 01:06:26.242185 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:26.242152 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-br7vh_5046e73d-17d6-4282-9387-c665c60f7ca9/serve-healthcheck-canary/0.log" Jul 21 01:06:26.879144 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:26.879113 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q5w7j_a7182242-ed71-49a7-ac63-fe6ced868d56/kube-rbac-proxy/0.log" Jul 21 01:06:26.901720 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:26.901697 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q5w7j_a7182242-ed71-49a7-ac63-fe6ced868d56/exporter/0.log" Jul 21 01:06:26.927690 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:26.927661 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q5w7j_a7182242-ed71-49a7-ac63-fe6ced868d56/extractor/0.log" Jul 21 01:06:27.736063 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:27.736035 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sf6l8/perf-node-gather-daemonset-rcqcb" Jul 21 01:06:29.218284 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:29.218253 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xpb5v_85364ca8-8d29-4a95-a39c-b9ea72ab6873/manager/0.log" Jul 21 01:06:29.483175 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:29.483116 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-s6fs8_01b985e5-79f1-4822-b3d5-93bf24f5599e/manager/0.log" Jul 21 01:06:29.631024 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:29.630991 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-79bfd5d9f6-d9ft9_d75f19e5-a207-4df5-a72e-9782ba3927a8/manager/0.log" Jul 21 01:06:29.671107 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:29.671069 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-2tln2_6a39002d-d7f1-4023-a125-2832e32db4f7/server/0.log" Jul 21 01:06:29.776329 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:29.776292 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-h24w6_8ce97c9b-8333-4464-a317-33fec0ef2813/manager/0.log" Jul 21 01:06:29.815138 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:29.815105 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7_49be6af6-766d-431d-b0cd-0963a5695e30/manager/0.log" Jul 21 01:06:29.856482 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:29.856455 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dfdb6d6cb-7rbb7_49be6af6-766d-431d-b0cd-0963a5695e30/copy-manifests/0.log" Jul 21 01:06:31.391684 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:31.391593 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-cpgp9_bb229d2f-6fc3-46a6-a331-cc469cd2587a/openshift-lws-operator/0.log" Jul 21 01:06:32.141840 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:32.141808 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4nzxzx_4d01be36-93d9-4180-90cf-ca8f7fde13b9/istio-proxy/0.log" Jul 21 01:06:32.274461 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:32.274432 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66766cb76c-plvdj_bf25e5c5-65e1-472c-b2d0-0623a79a7e4e/kube-auth-proxy/0.log" Jul 21 01:06:32.312420 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:32.312380 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-6vs2d_944990f8-eece-4bbe-a092-1db9b9547c2f/istio-proxy/0.log" Jul 21 01:06:32.350586 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:32.350490 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-6vs2d_944990f8-eece-4bbe-a092-1db9b9547c2f/istio-proxy/1.log" Jul 21 01:06:32.394991 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:32.394937 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-lxrfg_9f3e5dde-da98-4b74-aed1-d9351b3e325c/payload-processing/0.log" Jul 21 01:06:33.574715 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:33.574680 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4nzxzx_4d01be36-93d9-4180-90cf-ca8f7fde13b9/istio-proxy/0.log" Jul 21 01:06:33.708182 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:33.708149 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66766cb76c-plvdj_bf25e5c5-65e1-472c-b2d0-0623a79a7e4e/kube-auth-proxy/0.log" Jul 21 01:06:33.749129 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:33.749097 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-6vs2d_944990f8-eece-4bbe-a092-1db9b9547c2f/istio-proxy/0.log" Jul 21 01:06:33.786284 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:33.786254 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-6vs2d_944990f8-eece-4bbe-a092-1db9b9547c2f/istio-proxy/1.log" Jul 21 01:06:33.835403 ip-10-0-139-117 kubenswrapper[2580]: I0721 01:06:33.835303 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-lxrfg_9f3e5dde-da98-4b74-aed1-d9351b3e325c/payload-processing/0.log"