Nov 03 17:42:35.831742 ip-10-0-137-164 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 17:42:35.831751 ip-10-0-137-164 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 17:42:35.831758 ip-10-0-137-164 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 17:42:35.831977 ip-10-0-137-164 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 17:42:45.979859 ip-10-0-137-164 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 17:42:45.979873 ip-10-0-137-164 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 24add276db744d25aab0703b065e9c43 -- Nov 03 17:44:49.522677 ip-10-0-137-164 systemd[1]: Starting Kubernetes Kubelet... Nov 03 17:44:50.000610 ip-10-0-137-164 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 17:44:50.000610 ip-10-0-137-164 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 17:44:50.000610 ip-10-0-137-164 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 17:44:50.000610 ip-10-0-137-164 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 17:44:50.004035 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.003956 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:44:50.004035 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004036 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:44:50.004103 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004040 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:44:50.004103 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004043 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:44:50.004103 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004046 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:44:50.004103 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004049 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:44:50.004103 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004052 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:44:50.004103 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004054 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:44:50.004103 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004057 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:44:50.004103 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004059 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:44:50.004103 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004062 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:44:50.004103 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004065 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:44:50.004103 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004068 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:44:50.004103 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004071 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:44:50.004103 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004073 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:44:50.004103 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004077 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:44:50.004103 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004079 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:44:50.004103 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004083 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:44:50.004103 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004086 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:44:50.004103 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004088 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:44:50.004103 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004091 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:44:50.004550 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004093 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:44:50.004550 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004096 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:44:50.004550 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004099 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:44:50.004550 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004101 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:44:50.004550 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004104 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:44:50.004550 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004107 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:44:50.004550 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004110 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:44:50.004550 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004114 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:44:50.004550 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004119 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:44:50.004550 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004123 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:44:50.004550 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004126 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:44:50.004550 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004129 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:44:50.004550 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004132 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:44:50.004550 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004135 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:44:50.004550 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004138 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:44:50.004550 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004140 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:44:50.004550 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004143 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:44:50.004550 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004145 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:44:50.004550 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004148 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:44:50.005009 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004150 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:44:50.005009 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004152 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:44:50.005009 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004155 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:44:50.005009 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004157 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:44:50.005009 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004160 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:44:50.005009 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004162 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:44:50.005009 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004165 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:44:50.005009 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004167 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:44:50.005009 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004170 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:44:50.005009 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004173 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:44:50.005009 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004175 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:44:50.005009 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004178 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:44:50.005009 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004181 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:44:50.005009 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004183 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:44:50.005009 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004186 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:44:50.005009 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004189 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:44:50.005009 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004191 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:44:50.005009 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004194 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:44:50.005009 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004197 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:44:50.005009 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004199 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:44:50.005497 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004202 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:44:50.005497 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004204 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:44:50.005497 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004207 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:44:50.005497 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004209 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:44:50.005497 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004212 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:44:50.005497 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004214 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:44:50.005497 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004217 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:44:50.005497 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004220 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:44:50.005497 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004222 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:44:50.005497 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004225 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:44:50.005497 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004228 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:44:50.005497 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004230 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:44:50.005497 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004233 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:44:50.005497 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004236 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:44:50.005497 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004238 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:44:50.005497 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004241 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:44:50.005497 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004243 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:44:50.005497 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004247 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:44:50.005497 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004249 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:44:50.005497 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004252 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:44:50.005974 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004254 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:44:50.005974 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004257 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:44:50.005974 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004260 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:44:50.005974 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004263 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:44:50.005974 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004266 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:44:50.005974 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004269 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:44:50.005974 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004272 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:44:50.005974 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004276 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:44:50.005974 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004279 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:44:50.005974 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004282 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:44:50.005974 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004285 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:44:50.005974 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004288 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:44:50.005974 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004290 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:44:50.005974 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004293 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:44:50.005974 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004295 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:44:50.005974 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004298 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:44:50.005974 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004301 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:44:50.005974 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004303 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:44:50.005974 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004306 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:44:50.006438 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004308 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:44:50.006438 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004311 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:44:50.006438 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004313 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:44:50.006438 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004316 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:44:50.006438 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004319 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:44:50.006438 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004321 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:44:50.006438 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.004325 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:44:50.008556 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008542 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:44:50.008556 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008555 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:44:50.008616 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008558 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:44:50.008616 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008562 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:44:50.008616 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008565 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:44:50.008616 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008568 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:44:50.008616 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008571 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:44:50.008616 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008574 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:44:50.008616 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008577 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:44:50.008616 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008580 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:44:50.008616 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008583 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:44:50.008616 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008585 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:44:50.008616 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008588 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:44:50.008616 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008590 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:44:50.008616 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008593 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:44:50.008616 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008595 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:44:50.008616 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008598 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:44:50.008616 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008601 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:44:50.008616 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008603 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:44:50.008616 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008605 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:44:50.008616 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008608 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:44:50.009067 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008611 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:44:50.009067 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008613 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:44:50.009067 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008615 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:44:50.009067 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008618 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:44:50.009067 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008621 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:44:50.009067 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008624 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:44:50.009067 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008627 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:44:50.009067 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008629 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:44:50.009067 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008632 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:44:50.009067 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008634 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:44:50.009067 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008637 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:44:50.009067 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008639 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:44:50.009067 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008642 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:44:50.009067 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008644 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:44:50.009067 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008647 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:44:50.009067 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008650 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:44:50.009067 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008653 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:44:50.009067 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008655 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:44:50.009067 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008658 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:44:50.009529 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008660 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:44:50.009529 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008663 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:44:50.009529 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008666 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:44:50.009529 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008668 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:44:50.009529 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008670 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:44:50.009529 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008673 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:44:50.009529 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008675 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:44:50.009529 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008678 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:44:50.009529 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008680 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:44:50.009529 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008683 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:44:50.009529 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008685 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:44:50.009529 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008688 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:44:50.009529 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008690 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:44:50.009529 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008693 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:44:50.009529 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008697 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:44:50.009529 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008699 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:44:50.009529 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008702 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:44:50.009529 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008705 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:44:50.009529 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008707 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:44:50.009998 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008710 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:44:50.009998 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008713 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:44:50.009998 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008715 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:44:50.009998 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008718 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:44:50.009998 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008720 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:44:50.009998 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008723 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:44:50.009998 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008725 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:44:50.009998 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008728 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:44:50.009998 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008730 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:44:50.009998 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008733 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:44:50.009998 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008736 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:44:50.009998 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008738 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:44:50.009998 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008741 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:44:50.009998 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008743 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:44:50.009998 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008746 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:44:50.009998 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008749 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:44:50.009998 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008752 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:44:50.009998 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008754 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:44:50.009998 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008757 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:44:50.009998 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008759 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:44:50.010481 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008762 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:44:50.010481 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008764 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:44:50.010481 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008766 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:44:50.010481 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008769 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:44:50.010481 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008772 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:44:50.010481 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008774 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:44:50.010481 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008777 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:44:50.010481 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008780 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:44:50.010481 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008783 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:44:50.010481 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008786 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:44:50.010481 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008789 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:44:50.010481 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008791 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:44:50.010481 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008794 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:44:50.010481 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008797 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:44:50.010481 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008799 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:44:50.010481 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008803 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:44:50.010481 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008807 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:44:50.010481 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008811 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:44:50.010481 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008815 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:44:50.010926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008818 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:44:50.010926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008821 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:44:50.010926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008823 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:44:50.010926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008826 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:44:50.010926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008829 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:44:50.010926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008831 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:44:50.010926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008834 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:44:50.010926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.008836 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:44:50.010926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009325 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:44:50.010926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009330 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:44:50.010926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009333 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:44:50.010926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009335 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:44:50.010926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009339 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:44:50.010926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009341 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:44:50.010926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009344 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:44:50.010926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009347 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:44:50.010926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009350 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:44:50.010926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009352 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:44:50.010926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009355 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:44:50.011454 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009357 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:44:50.011454 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009361 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:44:50.011454 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009364 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:44:50.011454 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009367 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:44:50.011454 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009369 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:44:50.011454 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009372 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:44:50.011454 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009375 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:44:50.011454 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009377 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:44:50.011454 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009380 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:44:50.011454 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009383 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:44:50.011454 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009385 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:44:50.011454 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009388 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:44:50.011454 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009390 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:44:50.011454 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009411 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:44:50.011454 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009416 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:44:50.011454 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009420 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:44:50.011454 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009424 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:44:50.011454 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009426 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:44:50.011454 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009429 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:44:50.011454 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009432 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:44:50.012187 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009434 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:44:50.012187 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009437 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:44:50.012187 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009443 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:44:50.012187 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009446 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:44:50.012187 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009449 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:44:50.012187 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009451 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:44:50.012187 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009455 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:44:50.012187 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009458 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:44:50.012187 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009460 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:44:50.012187 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009463 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:44:50.012187 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009465 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:44:50.012187 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009468 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:44:50.012187 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009470 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:44:50.012187 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009474 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:44:50.012187 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009476 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:44:50.012187 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009479 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:44:50.012187 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009482 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:44:50.012187 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009484 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:44:50.012187 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009487 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:44:50.012684 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009490 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:44:50.012684 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009492 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:44:50.012684 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009495 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:44:50.012684 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009498 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:44:50.012684 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009500 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:44:50.012684 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009503 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:44:50.012684 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009505 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:44:50.012684 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009508 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:44:50.012684 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009510 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:44:50.012684 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009513 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:44:50.012684 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009515 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:44:50.012684 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009518 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:44:50.012684 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009520 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:44:50.012684 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009523 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:44:50.012684 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009525 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:44:50.012684 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009528 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:44:50.012684 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009531 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:44:50.012684 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009535 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:44:50.012684 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009538 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:44:50.013137 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009542 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:44:50.013137 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009544 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:44:50.013137 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009547 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:44:50.013137 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009550 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:44:50.013137 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009552 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:44:50.013137 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009555 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:44:50.013137 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009558 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:44:50.013137 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009561 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:44:50.013137 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009564 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:44:50.013137 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009566 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:44:50.013137 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009569 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:44:50.013137 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009572 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:44:50.013137 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009574 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:44:50.013137 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009577 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:44:50.013137 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009579 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:44:50.013137 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009582 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:44:50.013137 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009584 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:44:50.013137 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009587 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:44:50.013137 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009589 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:44:50.013137 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009592 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:44:50.013651 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009594 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:44:50.013651 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009597 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:44:50.013651 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009599 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:44:50.013651 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009602 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:44:50.013651 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009604 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:44:50.013651 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009607 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:44:50.013651 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009609 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:44:50.013651 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009612 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:44:50.013651 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009614 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:44:50.013651 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009616 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:44:50.013651 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009622 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:44:50.013651 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009626 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:44:50.013651 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009629 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:44:50.013651 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009632 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:44:50.013651 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009635 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:44:50.013651 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009637 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:44:50.013651 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.009640 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:44:50.013651 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010727 2574 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 17:44:50.013651 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010737 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 17:44:50.013651 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010744 2574 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 17:44:50.014159 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010749 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 17:44:50.014159 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010754 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 17:44:50.014159 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010757 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 17:44:50.014159 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010762 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 17:44:50.014159 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010766 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 17:44:50.014159 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010769 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 17:44:50.014159 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010772 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 17:44:50.014159 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010776 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 17:44:50.014159 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010779 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 17:44:50.014159 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010783 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 17:44:50.014159 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010786 2574 flags.go:64] FLAG: --cgroup-root="" Nov 03 17:44:50.014159 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010789 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 17:44:50.014159 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010792 2574 flags.go:64] FLAG: --client-ca-file="" Nov 03 17:44:50.014159 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010796 2574 flags.go:64] FLAG: --cloud-provider="external" Nov 03 17:44:50.014159 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010799 2574 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 17:44:50.014159 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010804 2574 flags.go:64] FLAG: --cluster-domain="" Nov 03 17:44:50.014159 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010807 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 17:44:50.014159 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010810 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 17:44:50.014159 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010813 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 17:44:50.014159 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010817 2574 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 17:44:50.014159 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010821 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 17:44:50.014159 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010824 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 17:44:50.014159 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010827 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010832 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010835 2574 flags.go:64] FLAG: --contention-profiling="false" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010838 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010841 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010844 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010847 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010852 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010855 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010858 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010861 2574 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010863 2574 flags.go:64] FLAG: --enable-server="true" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010866 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010871 2574 flags.go:64] FLAG: --event-burst="100" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010874 2574 flags.go:64] FLAG: --event-qps="50" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010877 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010880 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010883 2574 flags.go:64] FLAG: --eviction-hard="" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010887 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010890 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010894 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010896 2574 flags.go:64] FLAG: --eviction-soft="" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010899 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010902 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010905 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 17:44:50.014839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010908 2574 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 17:44:50.015530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010911 2574 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 17:44:50.015530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010914 2574 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 17:44:50.015530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010917 2574 flags.go:64] FLAG: --feature-gates="" Nov 03 17:44:50.015530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010921 2574 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 17:44:50.015530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010924 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 17:44:50.015530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010927 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 17:44:50.015530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010930 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 17:44:50.015530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010934 2574 flags.go:64] FLAG: --healthz-port="10248" Nov 03 17:44:50.015530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010937 2574 flags.go:64] FLAG: --help="false" Nov 03 17:44:50.015530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010940 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:44:50.015530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010944 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 17:44:50.015530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010947 2574 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 17:44:50.015530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010950 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 17:44:50.015530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010953 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 17:44:50.015530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010957 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 17:44:50.015530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010959 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 17:44:50.015530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010962 2574 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 17:44:50.015530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010965 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 17:44:50.015530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010968 2574 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 17:44:50.015530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010971 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 17:44:50.015530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010974 2574 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 17:44:50.015530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010977 2574 flags.go:64] FLAG: --kube-reserved="" Nov 03 17:44:50.015530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010980 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 17:44:50.015530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010982 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010985 2574 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010988 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010991 2574 flags.go:64] FLAG: --lock-file="" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010994 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.010998 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011001 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011006 2574 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011009 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011012 2574 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011015 2574 flags.go:64] FLAG: --logging-format="text" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011018 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011021 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011024 2574 flags.go:64] FLAG: --manifest-url="" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011027 2574 flags.go:64] FLAG: --manifest-url-header="" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011032 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011035 2574 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011039 2574 flags.go:64] FLAG: --max-pods="110" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011042 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011045 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011048 2574 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011051 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011054 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011057 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011060 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 17:44:50.016099 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011068 2574 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011071 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011074 2574 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011077 2574 flags.go:64] FLAG: --pod-cidr="" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011080 2574 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011083 2574 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011092 2574 flags.go:64] FLAG: --pods-per-core="0" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011096 2574 flags.go:64] FLAG: --port="10250" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011099 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011102 2574 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-07ee146735c7f22e8" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011105 2574 flags.go:64] FLAG: --qos-reserved="" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011108 2574 flags.go:64] FLAG: --read-only-port="10255" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011111 2574 flags.go:64] FLAG: --register-node="true" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011114 2574 flags.go:64] FLAG: --register-with-taints="" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011119 2574 flags.go:64] FLAG: --registry-burst="10" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011121 2574 flags.go:64] FLAG: --registry-qps="5" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011124 2574 flags.go:64] FLAG: --reserved-cpus="" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011127 2574 flags.go:64] FLAG: --reserved-memory="" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011131 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011134 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011137 2574 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011140 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011143 2574 flags.go:64] FLAG: --runonce="false" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011146 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011149 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011152 2574 flags.go:64] FLAG: --seccomp-default="false" Nov 03 17:44:50.016701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011156 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011159 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011162 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011165 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011168 2574 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011171 2574 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011174 2574 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011177 2574 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011180 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011183 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011186 2574 flags.go:64] FLAG: --system-cgroups="" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011189 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011194 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011197 2574 flags.go:64] FLAG: --tls-cert-file="" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011200 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011205 2574 flags.go:64] FLAG: --tls-min-version="" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011210 2574 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011213 2574 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011216 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011219 2574 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011222 2574 flags.go:64] FLAG: --v="2" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011226 2574 flags.go:64] FLAG: --version="false" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011231 2574 flags.go:64] FLAG: --vmodule="" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011235 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.011238 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 17:44:50.017307 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013655 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:44:50.017898 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013670 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:44:50.017898 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013675 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:44:50.017898 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013678 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:44:50.017898 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013682 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:44:50.017898 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013685 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:44:50.017898 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013688 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:44:50.017898 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013690 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:44:50.017898 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013693 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:44:50.017898 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013696 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:44:50.017898 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013698 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:44:50.017898 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013703 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:44:50.017898 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013706 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:44:50.017898 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013708 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:44:50.017898 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013711 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:44:50.017898 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013713 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:44:50.017898 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013717 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:44:50.017898 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013719 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:44:50.017898 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013722 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:44:50.017898 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013725 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:44:50.017898 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013728 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:44:50.018368 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013730 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:44:50.018368 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013733 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:44:50.018368 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013736 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:44:50.018368 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013738 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:44:50.018368 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013741 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:44:50.018368 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013745 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:44:50.018368 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013748 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:44:50.018368 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013751 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:44:50.018368 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013753 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:44:50.018368 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013756 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:44:50.018368 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013758 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:44:50.018368 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013762 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:44:50.018368 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013765 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:44:50.018368 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013767 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:44:50.018368 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013772 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:44:50.018368 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013774 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:44:50.018368 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013777 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:44:50.018368 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013780 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:44:50.018368 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013784 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:44:50.018899 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013790 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:44:50.018899 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013793 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:44:50.018899 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013796 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:44:50.018899 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013798 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:44:50.018899 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013801 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:44:50.018899 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013804 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:44:50.018899 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013807 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:44:50.018899 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013809 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:44:50.018899 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013812 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:44:50.018899 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013815 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:44:50.018899 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013817 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:44:50.018899 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013820 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:44:50.018899 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013823 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:44:50.018899 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013825 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:44:50.018899 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013829 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:44:50.018899 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013833 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:44:50.018899 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013836 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:44:50.018899 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013840 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:44:50.018899 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013843 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:44:50.019352 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013846 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:44:50.019352 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013848 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:44:50.019352 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013852 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:44:50.019352 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013854 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:44:50.019352 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013857 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:44:50.019352 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013861 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:44:50.019352 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013863 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:44:50.019352 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013866 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:44:50.019352 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013869 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:44:50.019352 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013871 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:44:50.019352 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013874 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:44:50.019352 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013876 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:44:50.019352 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013879 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:44:50.019352 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013882 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:44:50.019352 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013884 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:44:50.019352 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013887 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:44:50.019352 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013889 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:44:50.019352 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013892 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:44:50.019352 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013894 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:44:50.019352 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013897 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:44:50.019879 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013899 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:44:50.019879 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013902 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:44:50.019879 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013904 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:44:50.019879 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013907 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:44:50.019879 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013910 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:44:50.019879 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013913 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:44:50.019879 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013915 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:44:50.019879 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013918 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:44:50.019879 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013921 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:44:50.019879 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013923 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:44:50.019879 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013926 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:44:50.019879 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013928 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:44:50.019879 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013931 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:44:50.019879 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013934 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:44:50.019879 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013936 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:44:50.019879 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013939 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:44:50.019879 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013941 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:44:50.019879 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013944 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:44:50.019879 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013947 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:44:50.020329 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013949 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:44:50.020329 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013952 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:44:50.020329 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013955 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:44:50.020329 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013957 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:44:50.020329 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013959 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:44:50.020329 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013962 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:44:50.020329 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013965 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:44:50.020329 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.013967 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:44:50.020329 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.013998 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 17:44:50.020329 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014043 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:44:50.020329 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014047 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:44:50.020329 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014051 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:44:50.020329 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014053 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:44:50.020329 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014056 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:44:50.020329 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014059 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:44:50.020329 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014062 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:44:50.020720 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014064 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:44:50.020720 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014067 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:44:50.020720 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014070 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:44:50.020720 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014072 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:44:50.020720 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014075 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:44:50.020720 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014077 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:44:50.020720 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014080 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:44:50.020720 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014083 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:44:50.020720 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014086 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:44:50.020720 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014089 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:44:50.020720 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014092 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:44:50.020720 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014094 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:44:50.020720 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014097 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:44:50.020720 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014099 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:44:50.020720 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014103 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:44:50.020720 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014105 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:44:50.020720 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014108 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:44:50.020720 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014110 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:44:50.020720 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014113 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:44:50.021163 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014116 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:44:50.021163 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014118 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:44:50.021163 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014121 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:44:50.021163 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014123 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:44:50.021163 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014125 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:44:50.021163 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014128 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:44:50.021163 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014130 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:44:50.021163 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014133 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:44:50.021163 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014135 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:44:50.021163 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014138 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:44:50.021163 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014140 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:44:50.021163 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014144 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:44:50.021163 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014148 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:44:50.021163 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014151 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:44:50.021163 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014154 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:44:50.021163 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014156 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:44:50.021163 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014159 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:44:50.021163 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014162 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:44:50.021163 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014165 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:44:50.021163 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014167 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:44:50.021645 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014170 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:44:50.021645 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014173 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:44:50.021645 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014176 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:44:50.021645 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014178 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:44:50.021645 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014180 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:44:50.021645 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014183 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:44:50.021645 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014187 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:44:50.021645 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014192 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:44:50.021645 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014195 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:44:50.021645 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014198 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:44:50.021645 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014201 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:44:50.021645 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014204 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:44:50.021645 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014206 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:44:50.021645 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014209 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:44:50.021645 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014212 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:44:50.021645 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014214 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:44:50.021645 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014216 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:44:50.021645 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014219 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:44:50.021645 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014222 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:44:50.022095 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014224 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:44:50.022095 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014226 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:44:50.022095 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014229 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:44:50.022095 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014232 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:44:50.022095 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014234 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:44:50.022095 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014236 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:44:50.022095 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014239 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:44:50.022095 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014241 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:44:50.022095 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014244 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:44:50.022095 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014246 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:44:50.022095 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014249 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:44:50.022095 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014252 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:44:50.022095 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014254 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:44:50.022095 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014257 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:44:50.022095 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014262 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:44:50.022095 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014265 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:44:50.022095 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014267 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:44:50.022095 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014270 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:44:50.022095 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014272 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:44:50.022095 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014275 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:44:50.022581 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014278 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:44:50.022581 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014281 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:44:50.022581 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014283 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:44:50.022581 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014286 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:44:50.022581 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014290 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:44:50.022581 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014292 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:44:50.022581 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014295 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:44:50.022581 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014298 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:44:50.022581 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014300 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:44:50.022581 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014303 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:44:50.022581 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014305 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:44:50.022581 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014308 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:44:50.022581 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014310 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:44:50.022581 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014313 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:44:50.022581 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014315 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:44:50.022581 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014318 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:44:50.022581 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014320 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:44:50.022581 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014323 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:44:50.022581 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014325 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:44:50.022581 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014328 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:44:50.023044 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.014331 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:44:50.023044 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.021229 2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 17:44:50.023044 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.021245 2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 17:44:50.023044 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.022757 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 17:44:50.023044 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.022771 2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 17:44:50.023044 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022860 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:44:50.023044 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022868 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:44:50.023044 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022876 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:44:50.023044 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022883 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:44:50.023044 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022890 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:44:50.023044 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022895 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:44:50.023044 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022900 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:44:50.023044 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022905 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:44:50.023044 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022910 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:44:50.023044 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022914 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:44:50.023044 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022920 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:44:50.023044 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022924 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:44:50.023044 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022929 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:44:50.023044 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022933 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:44:50.023044 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022936 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:44:50.023567 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022940 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:44:50.023567 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022945 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:44:50.023567 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022949 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:44:50.023567 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022953 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:44:50.023567 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022958 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:44:50.023567 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022962 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:44:50.023567 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022966 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:44:50.023567 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022972 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:44:50.023567 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022978 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:44:50.023567 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022983 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:44:50.023567 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022988 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:44:50.023567 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022992 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:44:50.023567 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.022996 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:44:50.023567 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023001 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:44:50.023567 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023008 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:44:50.023567 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023013 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:44:50.023567 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023018 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:44:50.023567 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023023 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:44:50.024002 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023027 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:44:50.024002 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023032 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:44:50.024002 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023037 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:44:50.024002 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023042 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:44:50.024002 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023048 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:44:50.024002 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023053 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:44:50.024002 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023058 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:44:50.024002 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023063 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:44:50.024002 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023067 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:44:50.024002 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023072 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:44:50.024002 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023076 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:44:50.024002 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023080 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:44:50.024002 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023083 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:44:50.024002 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023097 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:44:50.024002 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023101 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:44:50.024002 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023105 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:44:50.024002 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023110 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:44:50.024002 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023113 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:44:50.024002 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023117 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:44:50.024002 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023122 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:44:50.024503 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023126 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:44:50.024503 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023130 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:44:50.024503 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023133 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:44:50.024503 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023137 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:44:50.024503 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023141 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:44:50.024503 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023144 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:44:50.024503 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023148 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:44:50.024503 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023152 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:44:50.024503 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023157 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:44:50.024503 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023163 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:44:50.024503 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023167 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:44:50.024503 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023172 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:44:50.024503 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023176 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:44:50.024503 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023180 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:44:50.024503 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023184 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:44:50.024503 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023189 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:44:50.024503 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023193 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:44:50.024503 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023197 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:44:50.024503 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023201 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:44:50.024503 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023205 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:44:50.024955 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023209 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:44:50.024955 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023213 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:44:50.024955 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023216 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:44:50.024955 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023220 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:44:50.024955 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023224 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:44:50.024955 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023228 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:44:50.024955 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023232 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:44:50.024955 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023236 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:44:50.024955 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023240 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:44:50.024955 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023243 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:44:50.024955 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023247 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:44:50.024955 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023251 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:44:50.024955 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023256 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:44:50.024955 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023261 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:44:50.024955 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023266 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:44:50.024955 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023270 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:44:50.024955 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023274 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:44:50.024955 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023279 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:44:50.024955 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023283 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:44:50.024955 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023288 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:44:50.025554 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023293 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:44:50.025554 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023297 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:44:50.025554 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023302 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:44:50.025554 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023307 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:44:50.025554 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023311 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:44:50.025554 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023315 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:44:50.025554 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023319 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:44:50.025554 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023323 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:44:50.025554 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023327 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:44:50.025554 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023332 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:44:50.025554 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023336 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:44:50.025554 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023341 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:44:50.025554 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023345 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:44:50.025554 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.023408 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 17:44:50.025554 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023641 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:44:50.025554 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023650 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:44:50.025926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023654 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:44:50.025926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023659 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:44:50.025926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023663 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:44:50.025926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023668 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:44:50.025926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023672 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:44:50.025926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023677 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:44:50.025926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023682 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:44:50.025926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023687 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:44:50.025926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023693 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:44:50.025926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023697 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:44:50.025926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023702 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:44:50.025926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023707 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:44:50.025926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023712 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:44:50.025926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023718 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:44:50.025926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023723 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:44:50.025926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023728 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:44:50.025926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023733 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:44:50.025926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023739 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:44:50.025926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023744 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:44:50.025926 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023749 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:44:50.026458 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023754 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:44:50.026458 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023758 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:44:50.026458 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023762 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:44:50.026458 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023766 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:44:50.026458 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023771 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:44:50.026458 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023775 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:44:50.026458 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023779 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:44:50.026458 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023784 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:44:50.026458 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023788 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:44:50.026458 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023792 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:44:50.026458 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023797 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:44:50.026458 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023801 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:44:50.026458 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023805 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:44:50.026458 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023812 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:44:50.026458 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023819 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:44:50.026458 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023825 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:44:50.026458 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023830 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:44:50.026458 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023835 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:44:50.026458 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023840 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:44:50.026904 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023844 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:44:50.026904 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023850 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:44:50.026904 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023854 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:44:50.026904 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023858 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:44:50.026904 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023865 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:44:50.026904 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023871 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:44:50.026904 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023875 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:44:50.026904 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023880 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:44:50.026904 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023885 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:44:50.026904 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023889 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:44:50.026904 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023893 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:44:50.026904 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023898 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:44:50.026904 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023902 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:44:50.026904 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023907 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:44:50.026904 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023911 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:44:50.026904 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023916 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:44:50.026904 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023920 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:44:50.026904 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023924 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:44:50.026904 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023928 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:44:50.026904 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023932 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:44:50.027404 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023936 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:44:50.027404 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023940 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:44:50.027404 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023945 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:44:50.027404 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023949 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:44:50.027404 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023953 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:44:50.027404 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023957 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:44:50.027404 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023961 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:44:50.027404 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023965 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:44:50.027404 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023970 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:44:50.027404 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023974 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:44:50.027404 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023978 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:44:50.027404 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023983 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:44:50.027404 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023986 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:44:50.027404 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023991 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:44:50.027404 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023995 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:44:50.027404 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.023999 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:44:50.027404 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024004 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:44:50.027404 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024008 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:44:50.027404 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024012 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:44:50.027404 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024017 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:44:50.027906 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024022 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:44:50.027906 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024027 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:44:50.027906 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024032 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:44:50.027906 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024036 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:44:50.027906 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024040 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:44:50.027906 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024045 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:44:50.027906 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024052 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:44:50.027906 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024057 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:44:50.027906 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024062 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:44:50.027906 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024068 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:44:50.027906 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024073 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:44:50.027906 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024077 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:44:50.027906 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024082 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:44:50.027906 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024087 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:44:50.027906 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024091 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:44:50.027906 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024095 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:44:50.027906 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024099 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:44:50.027906 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024103 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:44:50.027906 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024108 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:44:50.027906 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024112 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:44:50.028373 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024116 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:44:50.028373 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024121 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:44:50.028373 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024125 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:44:50.028373 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024130 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:44:50.028373 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024134 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:44:50.028373 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.024175 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 17:44:50.028373 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024227 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:44:50.028373 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024233 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:44:50.028373 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024237 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:44:50.028373 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024242 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:44:50.028373 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024246 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:44:50.028373 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024251 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:44:50.028373 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024257 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:44:50.028373 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024261 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:44:50.028373 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024266 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:44:50.028373 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024270 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:44:50.028748 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024275 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:44:50.028748 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024279 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:44:50.028748 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024284 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:44:50.028748 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024291 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:44:50.028748 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024296 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:44:50.028748 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024301 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:44:50.028748 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024306 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:44:50.028748 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024311 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:44:50.028748 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024315 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:44:50.028748 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024320 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:44:50.028748 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024324 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:44:50.028748 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024328 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:44:50.028748 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024332 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:44:50.028748 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024337 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:44:50.028748 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024341 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:44:50.028748 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024345 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:44:50.028748 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024349 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:44:50.028748 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024354 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:44:50.028748 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024358 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:44:50.028748 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024362 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:44:50.029223 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024366 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:44:50.029223 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024371 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:44:50.029223 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024375 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:44:50.029223 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024379 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:44:50.029223 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024386 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:44:50.029223 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024408 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:44:50.029223 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024414 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:44:50.029223 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024419 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:44:50.029223 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024424 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:44:50.029223 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024428 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:44:50.029223 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024432 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:44:50.029223 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024436 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:44:50.029223 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024441 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:44:50.029223 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024445 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:44:50.029223 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024451 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:44:50.029223 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024457 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:44:50.029223 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024464 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:44:50.029223 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024468 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:44:50.029223 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024472 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:44:50.029688 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024476 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:44:50.029688 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024481 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:44:50.029688 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024485 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:44:50.029688 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024489 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:44:50.029688 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024493 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:44:50.029688 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024497 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:44:50.029688 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024502 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:44:50.029688 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024506 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:44:50.029688 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024510 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:44:50.029688 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024515 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:44:50.029688 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024520 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:44:50.029688 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024524 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:44:50.029688 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024528 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:44:50.029688 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024533 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:44:50.029688 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024537 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:44:50.029688 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024541 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:44:50.029688 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024544 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:44:50.029688 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024548 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:44:50.029688 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024552 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:44:50.030129 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024556 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:44:50.030129 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024560 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:44:50.030129 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024564 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:44:50.030129 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024567 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:44:50.030129 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024571 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:44:50.030129 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024575 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:44:50.030129 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024579 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:44:50.030129 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024583 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:44:50.030129 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024587 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:44:50.030129 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024592 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:44:50.030129 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024596 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:44:50.030129 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024600 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:44:50.030129 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024604 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:44:50.030129 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024608 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:44:50.030129 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024612 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:44:50.030129 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024616 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:44:50.030129 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024620 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:44:50.030129 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024625 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:44:50.030129 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024629 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:44:50.030129 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024632 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:44:50.030668 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024636 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:44:50.030668 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024639 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:44:50.030668 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024643 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:44:50.030668 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024647 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:44:50.030668 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024652 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:44:50.030668 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024656 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:44:50.030668 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024660 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:44:50.030668 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024664 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:44:50.030668 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024668 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:44:50.030668 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024672 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:44:50.030668 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024675 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:44:50.030668 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024679 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:44:50.030668 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024683 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:44:50.030668 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024688 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:44:50.030668 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024692 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:44:50.030668 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024696 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:44:50.030668 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024700 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:44:50.030668 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:50.024707 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:44:50.030668 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.024804 2574 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 17:44:50.031153 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.027102 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 17:44:50.031153 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.028174 2574 server.go:1014] "Starting client certificate rotation" Nov 03 17:44:50.031153 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.028269 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 17:44:50.031153 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.028298 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 17:44:50.031344 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.031331 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 17:44:50.036146 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.036129 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 17:44:50.048936 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.048918 2574 log.go:25] "Validated CRI v1 runtime API" Nov 03 17:44:50.052417 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.052387 2574 log.go:25] "Validated CRI v1 image API" Nov 03 17:44:50.053253 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.053230 2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 17:44:50.061505 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.061479 2574 fs.go:135] Filesystem UUIDs: map[01eaba44-e3ba-432a-9ee3-8de92be6863c:/dev/nvme0n1p3 70390cd0-9b7f-404a-a0b9-f5b6eeed78c0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 03 17:44:50.061565 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.061502 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 17:44:50.067297 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.067189 2574 manager.go:217] Machine: {Timestamp:2026-11-03 17:44:50.065030504 +0000 UTC m=+0.409396872 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3178720 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22447227635aa58c11e2e0dbb1a41e SystemUUID:ec224472-2763-5aa5-8c11-e2e0dbb1a41e BootID:24add276-db74-4d25-aab0-703b065e9c43 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:cb:28:2a:e1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:cb:28:2a:e1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:5a:68:9e:e3:4f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 17:44:50.067377 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.067299 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 17:44:50.067377 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.067369 2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 17:44:50.071429 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.071410 2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 17:44:50.071554 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.071431 2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-164.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 17:44:50.071599 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.071556 2574 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 17:44:50.071599 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.071565 2574 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 17:44:50.071599 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.071590 2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 17:44:50.072535 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.072524 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 17:44:50.073712 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.073703 2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 17:44:50.077803 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.077790 2574 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 17:44:50.077915 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.077905 2574 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 17:44:50.078963 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.078954 2574 kubelet.go:493] "Attempting to sync node with API server" Nov 03 17:44:50.078995 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.078967 2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 17:44:50.078995 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.078979 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 17:44:50.078995 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.078988 2574 kubelet.go:405] "Adding apiserver pod source" Nov 03 17:44:50.079101 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.079003 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 17:44:50.079342 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.079321 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 17:44:50.082353 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.082337 2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 17:44:50.088314 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.088300 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 17:44:50.089303 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.089291 2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 17:44:50.089342 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.089326 2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 17:44:50.090067 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.090055 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 17:44:50.090115 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.090072 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 17:44:50.090115 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.090079 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 17:44:50.090115 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.090085 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 17:44:50.090115 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.090091 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 17:44:50.090115 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.090098 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 17:44:50.090466 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.090455 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 17:44:50.090466 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.090466 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 17:44:50.090525 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.090473 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 17:44:50.090525 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.090478 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 17:44:50.090525 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.090493 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 17:44:50.090525 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.090503 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 17:44:50.092618 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.092608 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 17:44:50.092618 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.092618 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 17:44:50.096851 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.096834 2574 server.go:1263] "Started kubelet" Nov 03 17:44:50.096945 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.096906 2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 17:44:50.097013 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.096961 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 17:44:50.097054 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.097038 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 17:44:50.097622 ip-10-0-137-164 systemd[1]: Started Kubernetes Kubelet. Nov 03 17:44:50.103130 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.103103 2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 17:44:50.104825 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:50.104778 2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 17:44:50.104939 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.104909 2574 server.go:317] "Adding debug handlers to kubelet server" Nov 03 17:44:50.109504 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.109118 2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 17:44:50.109504 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.109141 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 17:44:50.109504 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:50.109220 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-164.us-west-2.compute.internal\" not found" Nov 03 17:44:50.110384 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.109932 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 17:44:50.110384 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.109979 2574 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 17:44:50.110384 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.109997 2574 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 17:44:50.110384 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.110025 2574 factory.go:55] Registering systemd factory Nov 03 17:44:50.110384 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.110038 2574 factory.go:223] Registration of the systemd container factory successfully Nov 03 17:44:50.110384 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.110063 2574 reconstruct.go:97] "Volume reconstruction finished" Nov 03 17:44:50.110384 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.110073 2574 reconciler.go:29] "Reconciler: start to sync state" Nov 03 17:44:50.110384 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.110287 2574 factory.go:153] Registering CRI-O factory Nov 03 17:44:50.110384 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.110303 2574 factory.go:223] Registration of the crio container factory successfully Nov 03 17:44:50.110384 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.110356 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 17:44:50.110384 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.110378 2574 factory.go:103] Registering Raw factory Nov 03 17:44:50.110384 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.110409 2574 manager.go:1199] Started watching for new ooms in manager Nov 03 17:44:50.111388 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.111374 2574 manager.go:319] Starting recovery of all containers Nov 03 17:44:50.122616 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.122587 2574 manager.go:324] Recovery completed Nov 03 17:44:50.127450 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.127435 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:44:50.129133 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.129115 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:44:50.129219 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.129144 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:44:50.129219 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.129155 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:44:50.129690 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.129678 2574 cpu_manager.go:226] "Starting" policy="none" Nov 03 17:44:50.129763 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.129689 2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 17:44:50.129763 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.129678 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:44:50.129763 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.129708 2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 17:44:50.130347 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.130332 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:44:50.130488 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.130358 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:44:50.130488 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.130369 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:44:50.131790 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.131776 2574 policy_none.go:50] "Start" Nov 03 17:44:50.131790 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.131792 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 17:44:50.131877 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.131802 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 17:44:50.133776 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.133765 2574 policy_none.go:44] "Start" Nov 03 17:44:50.185581 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.177726 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 17:44:50.185581 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.178796 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 17:44:50.185581 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.178823 2574 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 17:44:50.185581 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.178849 2574 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 17:44:50.185581 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:50.178892 2574 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 17:44:50.185581 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.180112 2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 17:44:50.185581 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.182725 2574 manager.go:341] "Starting Device Plugin manager" Nov 03 17:44:50.185581 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:50.182809 2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 17:44:50.185581 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.182821 2574 server.go:87] "Starting device plugin registration server" Nov 03 17:44:50.185581 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.183056 2574 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 17:44:50.185581 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.183068 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 17:44:50.185581 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.183429 2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 17:44:50.185581 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.183496 2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 17:44:50.185581 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.183503 2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 17:44:50.185581 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:50.183959 2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 17:44:50.185581 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:50.184000 2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-164.us-west-2.compute.internal\" not found" Nov 03 17:44:50.246315 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.246287 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-164.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 17:44:50.246315 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:50.246298 2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-164.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 17:44:50.247356 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:50.246290 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-164.us-west-2.compute.internal.18e49a0330eb2e2e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-164.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-164.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 17:44:50.096598574 +0000 UTC m=+0.440964944,LastTimestamp:2026-11-03 17:44:50.096598574 +0000 UTC m=+0.440964944,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-164.us-west-2.compute.internal,}" Nov 03 17:44:50.280134 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.279947 2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal"] Nov 03 17:44:50.280228 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.280152 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:44:50.281049 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.281034 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:44:50.281133 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.281061 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:44:50.281133 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.281071 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:44:50.283137 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.283124 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:44:50.283210 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.283195 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:44:50.283272 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.283257 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:44:50.283310 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.283289 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:44:50.283828 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.283809 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:44:50.283922 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.283829 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:44:50.283922 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.283841 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:44:50.283922 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.283855 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:44:50.284018 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.283855 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:44:50.284018 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.283984 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:44:50.284018 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.284006 2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:44:50.284100 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.283815 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:44:50.284100 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.284066 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:44:50.284100 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.284076 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:44:50.285805 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.285787 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:44:50.285869 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.285814 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:44:50.286496 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.286478 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:44:50.286582 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.286510 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:44:50.286582 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.286520 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:44:50.311457 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.311430 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/79621dd6debcc143ca37704603624014-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal\" (UID: \"79621dd6debcc143ca37704603624014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:44:50.311457 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.311454 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79621dd6debcc143ca37704603624014-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal\" (UID: \"79621dd6debcc143ca37704603624014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:44:50.311635 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.311473 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5df37dbe21c11f3ff8879a1693476b0b-config\") pod \"kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal\" (UID: \"5df37dbe21c11f3ff8879a1693476b0b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:44:50.312620 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:50.312603 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-164.us-west-2.compute.internal\" not found" node="ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:44:50.315338 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:50.315323 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-164.us-west-2.compute.internal\" not found" node="ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:44:50.347197 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.347177 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 17:44:50.347271 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.347230 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l2jhf" Nov 03 17:44:50.357914 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:50.357885 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 17:44:50.358005 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:50.357846 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-164.us-west-2.compute.internal.18e49a0332db93c7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-164.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-164.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-164.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 17:44:50.129130439 +0000 UTC m=+0.473496810,LastTimestamp:2026-11-03 17:44:50.129130439 +0000 UTC m=+0.473496810,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-164.us-west-2.compute.internal,}" Nov 03 17:44:50.358005 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:50.357995 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 03 17:44:50.358113 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:50.358006 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-164.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 17:44:50.358149 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:50.358129 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 17:44:50.412262 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.412236 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/79621dd6debcc143ca37704603624014-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal\" (UID: \"79621dd6debcc143ca37704603624014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:44:50.412262 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.412262 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79621dd6debcc143ca37704603624014-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal\" (UID: \"79621dd6debcc143ca37704603624014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:44:50.412416 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.412279 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5df37dbe21c11f3ff8879a1693476b0b-config\") pod \"kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal\" (UID: \"5df37dbe21c11f3ff8879a1693476b0b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:44:50.412416 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.412332 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5df37dbe21c11f3ff8879a1693476b0b-config\") pod \"kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal\" (UID: \"5df37dbe21c11f3ff8879a1693476b0b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:44:50.412416 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.412343 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79621dd6debcc143ca37704603624014-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal\" (UID: \"79621dd6debcc143ca37704603624014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:44:50.412416 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.412332 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/79621dd6debcc143ca37704603624014-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal\" (UID: \"79621dd6debcc143ca37704603624014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:44:50.412902 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:50.412886 2574 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-137-164.us-west-2.compute.internal\" not found" node="ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:44:50.554900 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:50.554831 2574 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-164.us-west-2.compute.internal\" not found" node="ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:44:50.613696 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.613675 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:44:50.614670 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.614654 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:44:50.614763 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.614681 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:44:50.614763 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.614691 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:44:50.614763 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.614746 2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:44:50.616794 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.616780 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:44:50.620418 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.620389 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:44:50.672126 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:50.672114 2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:44:50.672198 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:50.672132 2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-164.us-west-2.compute.internal\": node \"ip-10-0-137-164.us-west-2.compute.internal\" not found" Nov 03 17:44:50.783131 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:50.783100 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-164.us-west-2.compute.internal\" not found" Nov 03 17:44:50.883522 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:50.883469 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-164.us-west-2.compute.internal\" not found" Nov 03 17:44:50.983957 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:50.983924 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-164.us-west-2.compute.internal\" not found" Nov 03 17:44:51.027655 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:51.027635 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 17:44:51.028072 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:51.027767 2574 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ad185ca6b177f4c7286538684ef3f4ce-d7e05c9bd6f41af3.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-137-164.us-west-2.compute.internal.18e49a0332dbddb8\": read tcp 10.0.137.164:38858->77.112.92.116:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-137-164.us-west-2.compute.internal.18e49a0332dbddb8 default 5234 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-164.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-164.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-164.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 17:44:50 +0000 UTC,LastTimestamp:2026-11-03 17:44:50.283847632 +0000 UTC m=+0.628214000,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-164.us-west-2.compute.internal,}" Nov 03 17:44:51.084988 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:51.084949 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-164.us-west-2.compute.internal\" not found" Nov 03 17:44:51.185233 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:51.185175 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-164.us-west-2.compute.internal\" not found" Nov 03 17:44:51.285963 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:51.285935 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-164.us-west-2.compute.internal\" not found" Nov 03 17:44:51.352226 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:51.352194 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 17:39:49 +0000 UTC" deadline="2028-07-04 03:55:43.380309354 +0000 UTC" Nov 03 17:44:51.352226 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:51.352221 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14602h10m52.028091649s" Nov 03 17:44:51.360279 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:51.360267 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:44:51.361151 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:51.361128 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:44:51.361241 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:51.361161 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:44:51.361241 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:51.361175 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:44:51.386456 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:51.386436 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-164.us-west-2.compute.internal\" not found" Nov 03 17:44:51.396522 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:51.396505 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:44:51.486616 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:51.486551 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-164.us-west-2.compute.internal\" not found" Nov 03 17:44:51.498077 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:51.498057 2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:44:51.587157 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:51.587131 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-164.us-west-2.compute.internal\" not found" Nov 03 17:44:51.673018 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:51.672986 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df37dbe21c11f3ff8879a1693476b0b.slice/crio-4357fa06359bdb104dcf16e83dd04927773998373e48cd06ac23b95f7e7441fb WatchSource:0}: Error finding container 4357fa06359bdb104dcf16e83dd04927773998373e48cd06ac23b95f7e7441fb: Status 404 returned error can't find the container with id 4357fa06359bdb104dcf16e83dd04927773998373e48cd06ac23b95f7e7441fb Nov 03 17:44:51.673215 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:51.673198 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79621dd6debcc143ca37704603624014.slice/crio-813fa771298c166a6343b5651245a1ed32536335e2e63b6e357d6f4210b18c70 WatchSource:0}: Error finding container 813fa771298c166a6343b5651245a1ed32536335e2e63b6e357d6f4210b18c70: Status 404 returned error can't find the container with id 813fa771298c166a6343b5651245a1ed32536335e2e63b6e357d6f4210b18c70 Nov 03 17:44:51.677453 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:51.677440 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 17:44:51.687781 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:51.687764 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-164.us-west-2.compute.internal\" not found" Nov 03 17:44:51.740888 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:51.740838 2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:44:51.748198 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:51.748176 2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:44:51.810179 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:51.810159 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:44:51.866581 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:51.866563 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 17:44:51.868464 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:51.868453 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:44:51.922193 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:51.922174 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 17:44:52.080564 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.080501 2574 apiserver.go:52] "Watching apiserver" Nov 03 17:44:52.110465 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.110446 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 17:44:52.175470 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.175448 2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 17:44:52.175709 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.175694 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr","openshift-multus/multus-additional-cni-plugins-fwttb","openshift-network-diagnostics/network-check-target-8p8q8","kube-system/konnectivity-agent-8b7gg","openshift-image-registry/node-ca-xk8dp","openshift-multus/network-metrics-daemon-fs77w","openshift-ovn-kubernetes/ovnkube-node-jjq2h","kube-system/global-pull-secret-syncer-8zpkh","openshift-cluster-node-tuning-operator/tuned-xm4q9","openshift-dns/node-resolver-qjbks","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal","openshift-multus/multus-4hhwj","openshift-network-operator/iptables-alerter-njt29","kube-system/kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal"] Nov 03 17:44:52.179262 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.179248 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.181007 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.180988 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-njt29" Nov 03 17:44:52.183025 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.182998 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:44:52.183113 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:52.183057 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p8q8" podUID="d2c0ef79-ea05-4061-b4dd-31edac551213" Nov 03 17:44:52.184934 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.184921 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8b7gg" Nov 03 17:44:52.186755 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.186740 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xk8dp" Nov 03 17:44:52.188687 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.188672 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:44:52.188754 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:52.188742 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs77w" podUID="0ba1ef12-3f5d-4449-8125-4a1921ec2990" Nov 03 17:44:52.190673 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.190660 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.192703 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.192686 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:44:52.194935 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.194919 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:44:52.199083 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.197305 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qjbks" Nov 03 17:44:52.200641 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.200621 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:44:52.200706 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:52.200679 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zpkh" podUID="c527a13f-da23-4228-a048-4af962c36340" Nov 03 17:44:52.202578 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.202546 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal" event={"ID":"5df37dbe21c11f3ff8879a1693476b0b","Type":"ContainerStarted","Data":"4357fa06359bdb104dcf16e83dd04927773998373e48cd06ac23b95f7e7441fb"} Nov 03 17:44:52.202732 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.202717 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.204730 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.204713 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" event={"ID":"79621dd6debcc143ca37704603624014","Type":"ContainerStarted","Data":"813fa771298c166a6343b5651245a1ed32536335e2e63b6e357d6f4210b18c70"} Nov 03 17:44:52.210606 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.210589 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 17:44:52.221776 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.221754 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:44:52.221841 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.221790 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-cnibin\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:44:52.221841 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.221833 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs\") pod \"network-metrics-daemon-fs77w\" (UID: \"0ba1ef12-3f5d-4449-8125-4a1921ec2990\") " pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:44:52.221899 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.221856 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2jll\" (UniqueName: \"kubernetes.io/projected/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-kube-api-access-t2jll\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.221899 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.221872 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret\") pod \"global-pull-secret-syncer-8zpkh\" (UID: \"c527a13f-da23-4228-a048-4af962c36340\") " pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:44:52.221964 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.221900 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93-hosts-file\") pod \"node-resolver-qjbks\" (UID: \"32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93\") " pod="openshift-dns/node-resolver-qjbks" Nov 03 17:44:52.221964 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.221943 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-run-ovn\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.222055 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.221989 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-cni-bin\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.222055 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222027 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-kubernetes\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.222148 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222072 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc524cbd-84c3-4255-a208-26fd55af457e-host-slash\") pod \"iptables-alerter-njt29\" (UID: \"dc524cbd-84c3-4255-a208-26fd55af457e\") " pod="openshift-network-operator/iptables-alerter-njt29" Nov 03 17:44:52.222148 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222096 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdzrj\" (UniqueName: \"kubernetes.io/projected/dc524cbd-84c3-4255-a208-26fd55af457e-kube-api-access-kdzrj\") pod \"iptables-alerter-njt29\" (UID: \"dc524cbd-84c3-4255-a208-26fd55af457e\") " pod="openshift-network-operator/iptables-alerter-njt29" Nov 03 17:44:52.222148 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222110 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-ovn-node-metrics-cert\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.222148 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222123 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-sysctl-d\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.222148 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222136 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-run-netns\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.222358 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222156 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-run\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.222358 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222173 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-sys\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.222358 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222186 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-var-lib-openvswitch\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.222358 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222252 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.222358 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222285 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-sysconfig\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.222358 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222309 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-host\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.222641 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222363 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jmjl\" (UniqueName: \"kubernetes.io/projected/0f7aa991-10ec-45ff-ac60-547adf0501a2-kube-api-access-8jmjl\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.222641 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222409 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:44:52.222641 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222434 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c527a13f-da23-4228-a048-4af962c36340-kubelet-config\") pod \"global-pull-secret-syncer-8zpkh\" (UID: \"c527a13f-da23-4228-a048-4af962c36340\") " pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:44:52.222641 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222454 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c527a13f-da23-4228-a048-4af962c36340-dbus\") pod \"global-pull-secret-syncer-8zpkh\" (UID: \"c527a13f-da23-4228-a048-4af962c36340\") " pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:44:52.222641 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222474 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-systemd-units\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.222641 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222516 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-run-systemd\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.222641 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222557 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-cni-netd\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.222641 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222580 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-ovnkube-config\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.222641 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222604 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:44:52.222641 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222628 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tv8z\" (UniqueName: \"kubernetes.io/projected/c9107231-c3e3-4d76-81e6-b276b828c9bc-kube-api-access-6tv8z\") pod \"node-ca-xk8dp\" (UID: \"c9107231-c3e3-4d76-81e6-b276b828c9bc\") " pod="openshift-image-registry/node-ca-xk8dp" Nov 03 17:44:52.223112 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222651 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gznl6\" (UniqueName: \"kubernetes.io/projected/0ba1ef12-3f5d-4449-8125-4a1921ec2990-kube-api-access-gznl6\") pod \"network-metrics-daemon-fs77w\" (UID: \"0ba1ef12-3f5d-4449-8125-4a1921ec2990\") " pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:44:52.223112 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222701 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-cnibin\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.223112 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222753 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-env-overrides\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.223112 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222800 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc524cbd-84c3-4255-a208-26fd55af457e-iptables-alerter-script\") pod \"iptables-alerter-njt29\" (UID: \"dc524cbd-84c3-4255-a208-26fd55af457e\") " pod="openshift-network-operator/iptables-alerter-njt29" Nov 03 17:44:52.223112 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222824 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f7aa991-10ec-45ff-ac60-547adf0501a2-tmp\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.223112 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222847 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-cni-binary-copy\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:44:52.223112 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222885 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8997w\" (UniqueName: \"kubernetes.io/projected/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-kube-api-access-8997w\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:44:52.223112 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222910 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93-tmp-dir\") pod \"node-resolver-qjbks\" (UID: \"32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93\") " pod="openshift-dns/node-resolver-qjbks" Nov 03 17:44:52.223112 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222947 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-multus-socket-dir-parent\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.223112 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.222983 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-run-k8s-cni-cncf-io\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.223112 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223012 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-sysctl-conf\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.223112 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223035 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-system-cni-dir\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.223112 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223060 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-multus-cni-dir\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.223112 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223084 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-var-lib-cni-bin\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.223112 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223106 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-var-lib-kubelet\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.223582 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223127 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-multus-conf-dir\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.223582 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223148 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/62507a50-2bff-4f45-82e7-aca45c032546-multus-daemon-config\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.223582 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223169 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-etc-kubernetes\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.223582 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223188 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-lib-modules\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.223582 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223209 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48n9p\" (UniqueName: \"kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p\") pod \"network-check-target-8p8q8\" (UID: \"d2c0ef79-ea05-4061-b4dd-31edac551213\") " pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:44:52.223582 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223229 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9107231-c3e3-4d76-81e6-b276b828c9bc-host\") pod \"node-ca-xk8dp\" (UID: \"c9107231-c3e3-4d76-81e6-b276b828c9bc\") " pod="openshift-image-registry/node-ca-xk8dp" Nov 03 17:44:52.223582 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223248 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62507a50-2bff-4f45-82e7-aca45c032546-cni-binary-copy\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.223582 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223268 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-run-netns\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.223582 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223287 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-hostroot\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.223582 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223309 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-modprobe-d\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.223582 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223331 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-socket-dir\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:44:52.223582 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223360 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-etc-selinux\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:44:52.223582 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223382 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-sys-fs\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:44:52.223582 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223436 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzmxx\" (UniqueName: \"kubernetes.io/projected/815e56fd-a11f-4a53-8d85-bfc073f33ea8-kube-api-access-lzmxx\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:44:52.223979 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223710 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:44:52.223979 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223762 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lcbs\" (UniqueName: \"kubernetes.io/projected/32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93-kube-api-access-6lcbs\") pod \"node-resolver-qjbks\" (UID: \"32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93\") " pod="openshift-dns/node-resolver-qjbks" Nov 03 17:44:52.223979 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223794 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-device-dir\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:44:52.223979 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223821 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa56a595-a9ef-431d-b176-41cb1cf382d7-agent-certs\") pod \"konnectivity-agent-8b7gg\" (UID: \"fa56a595-a9ef-431d-b176-41cb1cf382d7\") " pod="kube-system/konnectivity-agent-8b7gg" Nov 03 17:44:52.223979 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223848 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa56a595-a9ef-431d-b176-41cb1cf382d7-konnectivity-ca\") pod \"konnectivity-agent-8b7gg\" (UID: \"fa56a595-a9ef-431d-b176-41cb1cf382d7\") " pod="kube-system/konnectivity-agent-8b7gg" Nov 03 17:44:52.223979 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223872 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9107231-c3e3-4d76-81e6-b276b828c9bc-serviceca\") pod \"node-ca-xk8dp\" (UID: \"c9107231-c3e3-4d76-81e6-b276b828c9bc\") " pod="openshift-image-registry/node-ca-xk8dp" Nov 03 17:44:52.223979 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223888 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-var-lib-cni-multus\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.223979 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223902 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-etc-openvswitch\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.223979 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223915 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-ovnkube-script-lib\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.223979 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223929 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-system-cni-dir\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:44:52.223979 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223942 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-run-openvswitch\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.223979 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223956 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-run-ovn-kubernetes\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.223979 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.223981 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-node-log\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.224579 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.224009 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-kubelet\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.224579 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.224059 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-slash\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.224579 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.224078 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-log-socket\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.224579 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.224121 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-systemd\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.224579 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.224147 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-tuned\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.224579 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.224169 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-registration-dir\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:44:52.224579 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.224186 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-run-multus-certs\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.224579 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.224198 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6dcg\" (UniqueName: \"kubernetes.io/projected/62507a50-2bff-4f45-82e7-aca45c032546-kube-api-access-q6dcg\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.224579 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.224211 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-var-lib-kubelet\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.228218 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.228070 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 17:44:52.228218 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.228110 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 17:44:52.228218 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.228180 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 17:44:52.228218 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.228189 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 17:44:52.228440 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.228304 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 17:44:52.228440 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.228316 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8w5gn\"" Nov 03 17:44:52.228440 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.228434 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 17:44:52.229193 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.229177 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 17:44:52.229193 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.229176 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 17:44:52.229321 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.229209 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 17:44:52.229321 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.229234 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8lnxv\"" Nov 03 17:44:52.232977 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.232964 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 17:44:52.233813 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.233798 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-bmd8j\"" Nov 03 17:44:52.233813 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.233799 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 17:44:52.235539 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.235524 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 17:44:52.235603 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.235568 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 17:44:52.235843 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.235830 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 17:44:52.235890 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.235883 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l7l2n\"" Nov 03 17:44:52.239032 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.239021 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 17:44:52.239136 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.239125 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m6lsw\"" Nov 03 17:44:52.239167 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.239126 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 17:44:52.251864 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.251848 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 17:44:52.251946 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.251934 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 17:44:52.252299 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.252281 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 17:44:52.252433 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.252417 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 17:44:52.252493 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.252433 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 17:44:52.252493 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.252433 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mk8dz\"" Nov 03 17:44:52.252493 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.252420 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5hxz7\"" Nov 03 17:44:52.252493 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.252475 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z56rs\"" Nov 03 17:44:52.252643 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.252552 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-j4vc9\"" Nov 03 17:44:52.252643 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.252590 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 17:44:52.252963 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.252951 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 17:44:52.270850 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.270820 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 17:44:52.270850 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.270843 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 17:44:52.270850 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.270849 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 17:44:52.270968 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.270862 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 17:44:52.273119 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.273105 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 17:44:52.273159 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.273135 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-95jr7" Nov 03 17:44:52.325039 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325013 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/62507a50-2bff-4f45-82e7-aca45c032546-multus-daemon-config\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.325157 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325043 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-etc-kubernetes\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.325157 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325064 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-lib-modules\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.325157 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325083 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48n9p\" (UniqueName: \"kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p\") pod \"network-check-target-8p8q8\" (UID: \"d2c0ef79-ea05-4061-b4dd-31edac551213\") " pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:44:52.325157 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325101 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9107231-c3e3-4d76-81e6-b276b828c9bc-host\") pod \"node-ca-xk8dp\" (UID: \"c9107231-c3e3-4d76-81e6-b276b828c9bc\") " pod="openshift-image-registry/node-ca-xk8dp" Nov 03 17:44:52.325157 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325106 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-etc-kubernetes\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.325157 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325121 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62507a50-2bff-4f45-82e7-aca45c032546-cni-binary-copy\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.325157 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325140 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-run-netns\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.325157 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325146 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9107231-c3e3-4d76-81e6-b276b828c9bc-host\") pod \"node-ca-xk8dp\" (UID: \"c9107231-c3e3-4d76-81e6-b276b828c9bc\") " pod="openshift-image-registry/node-ca-xk8dp" Nov 03 17:44:52.325536 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325160 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-hostroot\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.325536 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325196 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-modprobe-d\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.325536 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325206 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-hostroot\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.325536 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325214 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-run-netns\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.325536 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325253 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-lib-modules\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.325536 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325220 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-socket-dir\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:44:52.325536 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325297 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-modprobe-d\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.325536 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325311 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-etc-selinux\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:44:52.325536 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325328 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-sys-fs\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:44:52.325536 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325341 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzmxx\" (UniqueName: \"kubernetes.io/projected/815e56fd-a11f-4a53-8d85-bfc073f33ea8-kube-api-access-lzmxx\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:44:52.325536 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325373 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-etc-selinux\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:44:52.325536 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325338 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-socket-dir\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:44:52.325536 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325416 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:44:52.325536 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325427 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-sys-fs\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:44:52.325536 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325439 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lcbs\" (UniqueName: \"kubernetes.io/projected/32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93-kube-api-access-6lcbs\") pod \"node-resolver-qjbks\" (UID: \"32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93\") " pod="openshift-dns/node-resolver-qjbks" Nov 03 17:44:52.325536 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325462 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-device-dir\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:44:52.325536 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325511 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa56a595-a9ef-431d-b176-41cb1cf382d7-agent-certs\") pod \"konnectivity-agent-8b7gg\" (UID: \"fa56a595-a9ef-431d-b176-41cb1cf382d7\") " pod="kube-system/konnectivity-agent-8b7gg" Nov 03 17:44:52.326206 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325536 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa56a595-a9ef-431d-b176-41cb1cf382d7-konnectivity-ca\") pod \"konnectivity-agent-8b7gg\" (UID: \"fa56a595-a9ef-431d-b176-41cb1cf382d7\") " pod="kube-system/konnectivity-agent-8b7gg" Nov 03 17:44:52.326206 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325549 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:44:52.326206 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325560 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9107231-c3e3-4d76-81e6-b276b828c9bc-serviceca\") pod \"node-ca-xk8dp\" (UID: \"c9107231-c3e3-4d76-81e6-b276b828c9bc\") " pod="openshift-image-registry/node-ca-xk8dp" Nov 03 17:44:52.326206 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325595 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-var-lib-cni-multus\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.326206 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325617 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-device-dir\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:44:52.326206 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325640 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-etc-openvswitch\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.326206 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325620 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-etc-openvswitch\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.326206 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325675 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-ovnkube-script-lib\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.326206 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325692 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62507a50-2bff-4f45-82e7-aca45c032546-cni-binary-copy\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.326206 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325697 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-system-cni-dir\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:44:52.326206 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325634 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/62507a50-2bff-4f45-82e7-aca45c032546-multus-daemon-config\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.326206 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325731 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-run-openvswitch\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.326206 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326038 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-run-openvswitch\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.326206 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326084 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-run-ovn-kubernetes\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.326206 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326130 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-node-log\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.326206 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326161 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-kubelet\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.326206 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326191 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-slash\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.326960 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326218 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-log-socket\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.326960 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326247 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-systemd\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.326960 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326279 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-tuned\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.326960 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326318 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-registration-dir\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:44:52.326960 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326357 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-run-multus-certs\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.326960 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326364 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa56a595-a9ef-431d-b176-41cb1cf382d7-konnectivity-ca\") pod \"konnectivity-agent-8b7gg\" (UID: \"fa56a595-a9ef-431d-b176-41cb1cf382d7\") " pod="kube-system/konnectivity-agent-8b7gg" Nov 03 17:44:52.326960 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326377 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9107231-c3e3-4d76-81e6-b276b828c9bc-serviceca\") pod \"node-ca-xk8dp\" (UID: \"c9107231-c3e3-4d76-81e6-b276b828c9bc\") " pod="openshift-image-registry/node-ca-xk8dp" Nov 03 17:44:52.326960 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326334 2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 17:44:52.326960 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326433 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-run-ovn-kubernetes\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.326960 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326428 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-slash\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.326960 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326502 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-node-log\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.326960 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326455 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-run-multus-certs\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.326960 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326507 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-registration-dir\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:44:52.326960 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326556 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-kubelet\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.326960 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326588 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-systemd\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.326960 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326597 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-log-socket\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.326960 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326597 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-ovnkube-script-lib\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.326960 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326613 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-system-cni-dir\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:44:52.327701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326644 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6dcg\" (UniqueName: \"kubernetes.io/projected/62507a50-2bff-4f45-82e7-aca45c032546-kube-api-access-q6dcg\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.327701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326672 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-var-lib-kubelet\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.327701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326699 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:44:52.327701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326729 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-cnibin\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:44:52.327701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326752 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-var-lib-kubelet\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.327701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326756 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs\") pod \"network-metrics-daemon-fs77w\" (UID: \"0ba1ef12-3f5d-4449-8125-4a1921ec2990\") " pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:44:52.327701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326817 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:44:52.327701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326823 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2jll\" (UniqueName: \"kubernetes.io/projected/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-kube-api-access-t2jll\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.327701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326819 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-cnibin\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:44:52.327701 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:52.326922 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:44:52.327701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.326974 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret\") pod \"global-pull-secret-syncer-8zpkh\" (UID: \"c527a13f-da23-4228-a048-4af962c36340\") " pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:44:52.327701 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:52.327013 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs podName:0ba1ef12-3f5d-4449-8125-4a1921ec2990 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:52.826991336 +0000 UTC m=+3.171357695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs") pod "network-metrics-daemon-fs77w" (UID: "0ba1ef12-3f5d-4449-8125-4a1921ec2990") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:44:52.327701 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:52.327031 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 17:44:52.327701 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:52.327073 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret podName:c527a13f-da23-4228-a048-4af962c36340 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:52.827058739 +0000 UTC m=+3.171425099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret") pod "global-pull-secret-syncer-8zpkh" (UID: "c527a13f-da23-4228-a048-4af962c36340") : object "kube-system"/"original-pull-secret" not registered Nov 03 17:44:52.327701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327102 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93-hosts-file\") pod \"node-resolver-qjbks\" (UID: \"32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93\") " pod="openshift-dns/node-resolver-qjbks" Nov 03 17:44:52.327701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327149 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93-hosts-file\") pod \"node-resolver-qjbks\" (UID: \"32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93\") " pod="openshift-dns/node-resolver-qjbks" Nov 03 17:44:52.327701 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327150 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-run-ovn\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.328467 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327174 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-run-ovn\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.328467 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327190 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-cni-bin\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.328467 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327230 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-cni-bin\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.328467 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327281 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-kubernetes\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.328467 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327309 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc524cbd-84c3-4255-a208-26fd55af457e-host-slash\") pod \"iptables-alerter-njt29\" (UID: \"dc524cbd-84c3-4255-a208-26fd55af457e\") " pod="openshift-network-operator/iptables-alerter-njt29" Nov 03 17:44:52.328467 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327337 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdzrj\" (UniqueName: \"kubernetes.io/projected/dc524cbd-84c3-4255-a208-26fd55af457e-kube-api-access-kdzrj\") pod \"iptables-alerter-njt29\" (UID: \"dc524cbd-84c3-4255-a208-26fd55af457e\") " pod="openshift-network-operator/iptables-alerter-njt29" Nov 03 17:44:52.328467 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327346 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-kubernetes\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.328467 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.325638 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-var-lib-cni-multus\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.328467 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327456 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc524cbd-84c3-4255-a208-26fd55af457e-host-slash\") pod \"iptables-alerter-njt29\" (UID: \"dc524cbd-84c3-4255-a208-26fd55af457e\") " pod="openshift-network-operator/iptables-alerter-njt29" Nov 03 17:44:52.328467 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327364 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-ovn-node-metrics-cert\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.328467 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327626 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-sysctl-d\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.328467 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327649 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-run-netns\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.328467 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327674 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-run\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.328467 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327700 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-sys\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.328467 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327730 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-sysctl-d\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.328467 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327737 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-var-lib-openvswitch\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.328467 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327774 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.329133 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327783 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-var-lib-openvswitch\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.329133 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327805 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-sysconfig\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.329133 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327828 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-run-netns\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.329133 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327829 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-host\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.329133 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327858 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-host\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.329133 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327865 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jmjl\" (UniqueName: \"kubernetes.io/projected/0f7aa991-10ec-45ff-ac60-547adf0501a2-kube-api-access-8jmjl\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.329133 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327903 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.329133 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327898 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:44:52.329133 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327944 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-sysconfig\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.329133 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327972 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c527a13f-da23-4228-a048-4af962c36340-kubelet-config\") pod \"global-pull-secret-syncer-8zpkh\" (UID: \"c527a13f-da23-4228-a048-4af962c36340\") " pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:44:52.329133 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.327989 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-sys\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.329133 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328010 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c527a13f-da23-4228-a048-4af962c36340-dbus\") pod \"global-pull-secret-syncer-8zpkh\" (UID: \"c527a13f-da23-4228-a048-4af962c36340\") " pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:44:52.329133 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328040 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-systemd-units\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.329133 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328050 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-run\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.329133 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328081 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-run-systemd\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.329133 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328116 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-cni-netd\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.329133 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328117 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c527a13f-da23-4228-a048-4af962c36340-kubelet-config\") pod \"global-pull-secret-syncer-8zpkh\" (UID: \"c527a13f-da23-4228-a048-4af962c36340\") " pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:44:52.329841 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328251 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c527a13f-da23-4228-a048-4af962c36340-dbus\") pod \"global-pull-secret-syncer-8zpkh\" (UID: \"c527a13f-da23-4228-a048-4af962c36340\") " pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:44:52.329841 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328333 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-run-systemd\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.329841 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328332 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-systemd-units\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.329841 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328407 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-cni-netd\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.329841 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328445 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-ovnkube-config\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.329841 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328474 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:44:52.329841 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328504 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tv8z\" (UniqueName: \"kubernetes.io/projected/c9107231-c3e3-4d76-81e6-b276b828c9bc-kube-api-access-6tv8z\") pod \"node-ca-xk8dp\" (UID: \"c9107231-c3e3-4d76-81e6-b276b828c9bc\") " pod="openshift-image-registry/node-ca-xk8dp" Nov 03 17:44:52.329841 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328531 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gznl6\" (UniqueName: \"kubernetes.io/projected/0ba1ef12-3f5d-4449-8125-4a1921ec2990-kube-api-access-gznl6\") pod \"network-metrics-daemon-fs77w\" (UID: \"0ba1ef12-3f5d-4449-8125-4a1921ec2990\") " pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:44:52.329841 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328555 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-cnibin\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.329841 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328579 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-env-overrides\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.329841 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328608 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc524cbd-84c3-4255-a208-26fd55af457e-iptables-alerter-script\") pod \"iptables-alerter-njt29\" (UID: \"dc524cbd-84c3-4255-a208-26fd55af457e\") " pod="openshift-network-operator/iptables-alerter-njt29" Nov 03 17:44:52.329841 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328633 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f7aa991-10ec-45ff-ac60-547adf0501a2-tmp\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.329841 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328658 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-cni-binary-copy\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:44:52.329841 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328684 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8997w\" (UniqueName: \"kubernetes.io/projected/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-kube-api-access-8997w\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:44:52.329841 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328706 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93-tmp-dir\") pod \"node-resolver-qjbks\" (UID: \"32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93\") " pod="openshift-dns/node-resolver-qjbks" Nov 03 17:44:52.329841 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328733 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-multus-socket-dir-parent\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.329841 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328759 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-run-k8s-cni-cncf-io\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.330355 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328785 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-sysctl-conf\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.330355 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328805 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-system-cni-dir\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.330355 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328829 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-multus-cni-dir\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.330355 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328853 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-var-lib-cni-bin\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.330355 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328890 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-var-lib-kubelet\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.330355 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328897 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:44:52.330355 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.328919 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-multus-conf-dir\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.330355 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.329403 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-ovnkube-config\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.330355 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.329423 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:44:52.330355 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.329632 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-multus-socket-dir-parent\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.330355 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.329685 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-run-k8s-cni-cncf-io\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.330355 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.329752 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-sysctl-conf\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.330355 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.329788 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-var-lib-kubelet\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.330355 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.329812 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93-tmp-dir\") pod \"node-resolver-qjbks\" (UID: \"32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93\") " pod="openshift-dns/node-resolver-qjbks" Nov 03 17:44:52.330355 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.329826 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-multus-cni-dir\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.330355 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.329889 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-var-lib-cni-bin\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.330355 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.330009 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-tuned\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.330355 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.330052 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-ovn-node-metrics-cert\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.331130 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.330208 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-cni-binary-copy\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:44:52.331130 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.330208 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa56a595-a9ef-431d-b176-41cb1cf382d7-agent-certs\") pod \"konnectivity-agent-8b7gg\" (UID: \"fa56a595-a9ef-431d-b176-41cb1cf382d7\") " pod="kube-system/konnectivity-agent-8b7gg" Nov 03 17:44:52.331130 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.330447 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-multus-conf-dir\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.331130 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.330574 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-env-overrides\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.331130 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.330623 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-cnibin\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.331130 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.330855 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-system-cni-dir\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.331368 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.331322 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc524cbd-84c3-4255-a208-26fd55af457e-iptables-alerter-script\") pod \"iptables-alerter-njt29\" (UID: \"dc524cbd-84c3-4255-a208-26fd55af457e\") " pod="openshift-network-operator/iptables-alerter-njt29" Nov 03 17:44:52.332373 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.332352 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f7aa991-10ec-45ff-ac60-547adf0501a2-tmp\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.379978 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:52.379956 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 17:44:52.379978 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:52.379978 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 17:44:52.380125 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:52.379990 2574 projected.go:196] Error preparing data for projected volume kube-api-access-48n9p for pod openshift-network-diagnostics/network-check-target-8p8q8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:44:52.380125 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:52.380059 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p podName:d2c0ef79-ea05-4061-b4dd-31edac551213 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:52.880043351 +0000 UTC m=+3.224409720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-48n9p" (UniqueName: "kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p") pod "network-check-target-8p8q8" (UID: "d2c0ef79-ea05-4061-b4dd-31edac551213") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:44:52.381653 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.381632 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzmxx\" (UniqueName: \"kubernetes.io/projected/815e56fd-a11f-4a53-8d85-bfc073f33ea8-kube-api-access-lzmxx\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:44:52.381820 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.381800 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lcbs\" (UniqueName: \"kubernetes.io/projected/32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93-kube-api-access-6lcbs\") pod \"node-resolver-qjbks\" (UID: \"32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93\") " pod="openshift-dns/node-resolver-qjbks" Nov 03 17:44:52.385692 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.385674 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6dcg\" (UniqueName: \"kubernetes.io/projected/62507a50-2bff-4f45-82e7-aca45c032546-kube-api-access-q6dcg\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.391176 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.391151 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2jll\" (UniqueName: \"kubernetes.io/projected/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-kube-api-access-t2jll\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.391699 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.391680 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tv8z\" (UniqueName: \"kubernetes.io/projected/c9107231-c3e3-4d76-81e6-b276b828c9bc-kube-api-access-6tv8z\") pod \"node-ca-xk8dp\" (UID: \"c9107231-c3e3-4d76-81e6-b276b828c9bc\") " pod="openshift-image-registry/node-ca-xk8dp" Nov 03 17:44:52.391891 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.391861 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jmjl\" (UniqueName: \"kubernetes.io/projected/0f7aa991-10ec-45ff-ac60-547adf0501a2-kube-api-access-8jmjl\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.391992 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.391978 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdzrj\" (UniqueName: \"kubernetes.io/projected/dc524cbd-84c3-4255-a208-26fd55af457e-kube-api-access-kdzrj\") pod \"iptables-alerter-njt29\" (UID: \"dc524cbd-84c3-4255-a208-26fd55af457e\") " pod="openshift-network-operator/iptables-alerter-njt29" Nov 03 17:44:52.392101 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.392086 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gznl6\" (UniqueName: \"kubernetes.io/projected/0ba1ef12-3f5d-4449-8125-4a1921ec2990-kube-api-access-gznl6\") pod \"network-metrics-daemon-fs77w\" (UID: \"0ba1ef12-3f5d-4449-8125-4a1921ec2990\") " pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:44:52.392101 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.392094 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8997w\" (UniqueName: \"kubernetes.io/projected/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-kube-api-access-8997w\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:44:52.501860 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.501826 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-njt29" Nov 03 17:44:52.503992 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.503959 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8b7gg" Nov 03 17:44:52.508587 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.508568 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:44:52.510638 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.510621 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xk8dp" Nov 03 17:44:52.512492 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.512475 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:44:52.517008 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:52.516917 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa56a595_a9ef_431d_b176_41cb1cf382d7.slice/crio-5d8b95e229d388e4131f73e09a14dc0be4f6872d1806b548cafc9ceb8b8e5900 WatchSource:0}: Error finding container 5d8b95e229d388e4131f73e09a14dc0be4f6872d1806b548cafc9ceb8b8e5900: Status 404 returned error can't find the container with id 5d8b95e229d388e4131f73e09a14dc0be4f6872d1806b548cafc9ceb8b8e5900 Nov 03 17:44:52.521196 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.520506 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:44:52.521196 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:52.520853 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a74bba5_bc9c_4ae1_9754_8887c5bf23cc.slice/crio-160ad685081be00afa353761e3e9725471a84deb4ed3ec1ea7e5f0daeddda5f0 WatchSource:0}: Error finding container 160ad685081be00afa353761e3e9725471a84deb4ed3ec1ea7e5f0daeddda5f0: Status 404 returned error can't find the container with id 160ad685081be00afa353761e3e9725471a84deb4ed3ec1ea7e5f0daeddda5f0 Nov 03 17:44:52.524185 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:52.524144 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9107231_c3e3_4d76_81e6_b276b828c9bc.slice/crio-e7ba28d804f716a25a76352467895bfb7569f8fb85ae1ae38a4c4242cd6a9c9b WatchSource:0}: Error finding container e7ba28d804f716a25a76352467895bfb7569f8fb85ae1ae38a4c4242cd6a9c9b: Status 404 returned error can't find the container with id e7ba28d804f716a25a76352467895bfb7569f8fb85ae1ae38a4c4242cd6a9c9b Nov 03 17:44:52.525935 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:52.525920 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f7aa991_10ec_45ff_ac60_547adf0501a2.slice/crio-2209c0178520a5ead78d28604d8cdedeea24ff4b7ba236a2e86ebef4e6d43f3f WatchSource:0}: Error finding container 2209c0178520a5ead78d28604d8cdedeea24ff4b7ba236a2e86ebef4e6d43f3f: Status 404 returned error can't find the container with id 2209c0178520a5ead78d28604d8cdedeea24ff4b7ba236a2e86ebef4e6d43f3f Nov 03 17:44:52.529865 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:52.529846 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod815e56fd_a11f_4a53_8d85_bfc073f33ea8.slice/crio-f6c1cef46b79d828e3a40b0616b5ef026150386d5da980d30f50b9969d0f0e8d WatchSource:0}: Error finding container f6c1cef46b79d828e3a40b0616b5ef026150386d5da980d30f50b9969d0f0e8d: Status 404 returned error can't find the container with id f6c1cef46b79d828e3a40b0616b5ef026150386d5da980d30f50b9969d0f0e8d Nov 03 17:44:52.531992 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.531973 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qjbks" Nov 03 17:44:52.534363 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.534343 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4hhwj" Nov 03 17:44:52.537850 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.537831 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:44:52.541085 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:52.541063 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32cb83db_b6ab_4d7e_ae38_2fb9b06e2c93.slice/crio-41b9d83a33d48f80a45e14cb54ed997cb98bc4861f5c04746ad518e117089c82 WatchSource:0}: Error finding container 41b9d83a33d48f80a45e14cb54ed997cb98bc4861f5c04746ad518e117089c82: Status 404 returned error can't find the container with id 41b9d83a33d48f80a45e14cb54ed997cb98bc4861f5c04746ad518e117089c82 Nov 03 17:44:52.543056 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:52.542900 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62507a50_2bff_4f45_82e7_aca45c032546.slice/crio-4b21927369d3f0d71c349f5d70434c29a3217f3ced7e85f5928a95c267285ab9 WatchSource:0}: Error finding container 4b21927369d3f0d71c349f5d70434c29a3217f3ced7e85f5928a95c267285ab9: Status 404 returned error can't find the container with id 4b21927369d3f0d71c349f5d70434c29a3217f3ced7e85f5928a95c267285ab9 Nov 03 17:44:52.548034 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:44:52.548009 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a9f32d_0e80_42ce_81ad_d4c08790fccb.slice/crio-5f96e593da76921ede9472c64a38ccc47f916b3cfa0e24a03ad6c00752045278 WatchSource:0}: Error finding container 5f96e593da76921ede9472c64a38ccc47f916b3cfa0e24a03ad6c00752045278: Status 404 returned error can't find the container with id 5f96e593da76921ede9472c64a38ccc47f916b3cfa0e24a03ad6c00752045278 Nov 03 17:44:52.831499 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.831472 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs\") pod \"network-metrics-daemon-fs77w\" (UID: \"0ba1ef12-3f5d-4449-8125-4a1921ec2990\") " pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:44:52.831659 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.831516 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret\") pod \"global-pull-secret-syncer-8zpkh\" (UID: \"c527a13f-da23-4228-a048-4af962c36340\") " pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:44:52.831659 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:52.831630 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:44:52.831659 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:52.831653 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 17:44:52.831780 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:52.831699 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs podName:0ba1ef12-3f5d-4449-8125-4a1921ec2990 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:53.831681684 +0000 UTC m=+4.176048038 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs") pod "network-metrics-daemon-fs77w" (UID: "0ba1ef12-3f5d-4449-8125-4a1921ec2990") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:44:52.831780 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:52.831713 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret podName:c527a13f-da23-4228-a048-4af962c36340 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:53.831706692 +0000 UTC m=+4.176073044 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret") pod "global-pull-secret-syncer-8zpkh" (UID: "c527a13f-da23-4228-a048-4af962c36340") : object "kube-system"/"original-pull-secret" not registered Nov 03 17:44:52.932745 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:52.932718 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48n9p\" (UniqueName: \"kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p\") pod \"network-check-target-8p8q8\" (UID: \"d2c0ef79-ea05-4061-b4dd-31edac551213\") " pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:44:52.932899 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:52.932880 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 17:44:52.932899 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:52.932896 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 17:44:52.932986 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:52.932907 2574 projected.go:196] Error preparing data for projected volume kube-api-access-48n9p for pod openshift-network-diagnostics/network-check-target-8p8q8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:44:52.932986 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:52.932958 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p podName:d2c0ef79-ea05-4061-b4dd-31edac551213 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:53.932946918 +0000 UTC m=+4.277313283 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-48n9p" (UniqueName: "kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p") pod "network-check-target-8p8q8" (UID: "d2c0ef79-ea05-4061-b4dd-31edac551213") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:44:53.187578 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:53.187440 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4hhwj" event={"ID":"62507a50-2bff-4f45-82e7-aca45c032546","Type":"ContainerStarted","Data":"4b21927369d3f0d71c349f5d70434c29a3217f3ced7e85f5928a95c267285ab9"} Nov 03 17:44:53.188708 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:53.188680 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" event={"ID":"0f7aa991-10ec-45ff-ac60-547adf0501a2","Type":"ContainerStarted","Data":"2209c0178520a5ead78d28604d8cdedeea24ff4b7ba236a2e86ebef4e6d43f3f"} Nov 03 17:44:53.190045 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:53.190016 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" event={"ID":"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc","Type":"ContainerStarted","Data":"160ad685081be00afa353761e3e9725471a84deb4ed3ec1ea7e5f0daeddda5f0"} Nov 03 17:44:53.191133 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:53.191115 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8b7gg" event={"ID":"fa56a595-a9ef-431d-b176-41cb1cf382d7","Type":"ContainerStarted","Data":"5d8b95e229d388e4131f73e09a14dc0be4f6872d1806b548cafc9ceb8b8e5900"} Nov 03 17:44:53.192442 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:53.192416 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-njt29" event={"ID":"dc524cbd-84c3-4255-a208-26fd55af457e","Type":"ContainerStarted","Data":"b6ef76c9ffbb6353f61b51f36eec06480fe19c75a27eb2a0bef3dda8b1051114"} Nov 03 17:44:53.193470 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:53.193450 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwttb" event={"ID":"d3a9f32d-0e80-42ce-81ad-d4c08790fccb","Type":"ContainerStarted","Data":"5f96e593da76921ede9472c64a38ccc47f916b3cfa0e24a03ad6c00752045278"} Nov 03 17:44:53.194517 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:53.194497 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qjbks" event={"ID":"32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93","Type":"ContainerStarted","Data":"41b9d83a33d48f80a45e14cb54ed997cb98bc4861f5c04746ad518e117089c82"} Nov 03 17:44:53.195685 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:53.195666 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" event={"ID":"815e56fd-a11f-4a53-8d85-bfc073f33ea8","Type":"ContainerStarted","Data":"f6c1cef46b79d828e3a40b0616b5ef026150386d5da980d30f50b9969d0f0e8d"} Nov 03 17:44:53.196908 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:53.196887 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xk8dp" event={"ID":"c9107231-c3e3-4d76-81e6-b276b828c9bc","Type":"ContainerStarted","Data":"e7ba28d804f716a25a76352467895bfb7569f8fb85ae1ae38a4c4242cd6a9c9b"} Nov 03 17:44:53.276363 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:53.276297 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 17:39:51 +0000 UTC" deadline="2028-08-07 20:30:21.929302914 +0000 UTC" Nov 03 17:44:53.276363 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:53.276358 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15434h45m28.652949469s" Nov 03 17:44:53.838498 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:53.838468 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs\") pod \"network-metrics-daemon-fs77w\" (UID: \"0ba1ef12-3f5d-4449-8125-4a1921ec2990\") " pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:44:53.838605 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:53.838513 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret\") pod \"global-pull-secret-syncer-8zpkh\" (UID: \"c527a13f-da23-4228-a048-4af962c36340\") " pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:44:53.838656 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:53.838622 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 17:44:53.838702 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:53.838673 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret podName:c527a13f-da23-4228-a048-4af962c36340 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:55.83865829 +0000 UTC m=+6.183024658 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret") pod "global-pull-secret-syncer-8zpkh" (UID: "c527a13f-da23-4228-a048-4af962c36340") : object "kube-system"/"original-pull-secret" not registered Nov 03 17:44:53.838771 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:53.838730 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:44:53.838828 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:53.838759 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs podName:0ba1ef12-3f5d-4449-8125-4a1921ec2990 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:55.838748568 +0000 UTC m=+6.183114924 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs") pod "network-metrics-daemon-fs77w" (UID: "0ba1ef12-3f5d-4449-8125-4a1921ec2990") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:44:53.940565 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:53.940287 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48n9p\" (UniqueName: \"kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p\") pod \"network-check-target-8p8q8\" (UID: \"d2c0ef79-ea05-4061-b4dd-31edac551213\") " pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:44:53.940565 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:53.940457 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 17:44:53.940565 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:53.940475 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 17:44:53.940565 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:53.940487 2574 projected.go:196] Error preparing data for projected volume kube-api-access-48n9p for pod openshift-network-diagnostics/network-check-target-8p8q8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:44:53.940565 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:53.940542 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p podName:d2c0ef79-ea05-4061-b4dd-31edac551213 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:55.940525999 +0000 UTC m=+6.284892361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-48n9p" (UniqueName: "kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p") pod "network-check-target-8p8q8" (UID: "d2c0ef79-ea05-4061-b4dd-31edac551213") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:44:54.180930 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:54.180860 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:44:54.181049 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:54.180983 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p8q8" podUID="d2c0ef79-ea05-4061-b4dd-31edac551213" Nov 03 17:44:54.181049 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:54.181027 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:44:54.181049 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:54.181047 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:44:54.181198 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:54.181128 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zpkh" podUID="c527a13f-da23-4228-a048-4af962c36340" Nov 03 17:44:54.181248 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:54.181218 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs77w" podUID="0ba1ef12-3f5d-4449-8125-4a1921ec2990" Nov 03 17:44:54.202671 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:54.202639 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal" event={"ID":"5df37dbe21c11f3ff8879a1693476b0b","Type":"ContainerStarted","Data":"d885c791a36ee714ebf6c5a7a2576b75becd09ec2562133570cb236c82269e7e"} Nov 03 17:44:54.278435 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:54.278361 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 17:39:51 +0000 UTC" deadline="2028-04-18 08:14:20.092124604 +0000 UTC" Nov 03 17:44:54.278435 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:54.278431 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12758h29m25.813697039s" Nov 03 17:44:54.311188 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:54.309628 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal" podStartSLOduration=3.309608941 podStartE2EDuration="3.309608941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:54.308931514 +0000 UTC m=+4.653297902" watchObservedRunningTime="2026-11-03 17:44:54.309608941 +0000 UTC m=+4.653975315" Nov 03 17:44:55.209505 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:55.209427 2574 generic.go:356] "Generic (PLEG): container finished" podID="79621dd6debcc143ca37704603624014" containerID="9480f947e0b0ddfc27d46853e2fc647f346bb07725948977c119f697e270f3ac" exitCode=0 Nov 03 17:44:55.210364 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:55.210339 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" event={"ID":"79621dd6debcc143ca37704603624014","Type":"ContainerDied","Data":"9480f947e0b0ddfc27d46853e2fc647f346bb07725948977c119f697e270f3ac"} Nov 03 17:44:55.857354 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:55.857324 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret\") pod \"global-pull-secret-syncer-8zpkh\" (UID: \"c527a13f-da23-4228-a048-4af962c36340\") " pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:44:55.857564 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:55.857461 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs\") pod \"network-metrics-daemon-fs77w\" (UID: \"0ba1ef12-3f5d-4449-8125-4a1921ec2990\") " pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:44:55.857564 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:55.857486 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 17:44:55.857564 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:55.857534 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:44:55.857564 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:55.857546 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret podName:c527a13f-da23-4228-a048-4af962c36340 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:59.85752914 +0000 UTC m=+10.201895515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret") pod "global-pull-secret-syncer-8zpkh" (UID: "c527a13f-da23-4228-a048-4af962c36340") : object "kube-system"/"original-pull-secret" not registered Nov 03 17:44:55.857564 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:55.857564 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs podName:0ba1ef12-3f5d-4449-8125-4a1921ec2990 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:59.857554232 +0000 UTC m=+10.201920587 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs") pod "network-metrics-daemon-fs77w" (UID: "0ba1ef12-3f5d-4449-8125-4a1921ec2990") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:44:55.958066 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:55.958033 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48n9p\" (UniqueName: \"kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p\") pod \"network-check-target-8p8q8\" (UID: \"d2c0ef79-ea05-4061-b4dd-31edac551213\") " pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:44:55.958209 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:55.958171 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 17:44:55.958209 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:55.958189 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 17:44:55.958209 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:55.958200 2574 projected.go:196] Error preparing data for projected volume kube-api-access-48n9p for pod openshift-network-diagnostics/network-check-target-8p8q8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:44:55.958380 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:55.958249 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p podName:d2c0ef79-ea05-4061-b4dd-31edac551213 nodeName:}" failed. No retries permitted until 2026-11-03 17:44:59.958233681 +0000 UTC m=+10.302600049 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-48n9p" (UniqueName: "kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p") pod "network-check-target-8p8q8" (UID: "d2c0ef79-ea05-4061-b4dd-31edac551213") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:44:56.179559 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:56.179487 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:44:56.179704 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:56.179618 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zpkh" podUID="c527a13f-da23-4228-a048-4af962c36340" Nov 03 17:44:56.179704 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:56.179674 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:44:56.179824 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:56.179737 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p8q8" podUID="d2c0ef79-ea05-4061-b4dd-31edac551213" Nov 03 17:44:56.181968 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:56.181938 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:44:56.182151 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:56.182048 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs77w" podUID="0ba1ef12-3f5d-4449-8125-4a1921ec2990" Nov 03 17:44:56.216173 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:56.216143 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" event={"ID":"79621dd6debcc143ca37704603624014","Type":"ContainerStarted","Data":"5d7f4f365c0af7bd9915fd75a3b5b87182707ba405a75d17dd28b1a8e89cdfbe"} Nov 03 17:44:56.219322 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:56.219293 2574 generic.go:356] "Generic (PLEG): container finished" podID="d3a9f32d-0e80-42ce-81ad-d4c08790fccb" containerID="08f89d11852613c79ae48fe38669f896b846334ef8eb23630f28d2fcf6a87094" exitCode=0 Nov 03 17:44:56.219481 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:56.219336 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwttb" event={"ID":"d3a9f32d-0e80-42ce-81ad-d4c08790fccb","Type":"ContainerDied","Data":"08f89d11852613c79ae48fe38669f896b846334ef8eb23630f28d2fcf6a87094"} Nov 03 17:44:56.455971 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:56.455842 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" podStartSLOduration=5.45582383 podStartE2EDuration="5.45582383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:56.333291896 +0000 UTC m=+6.677658293" watchObservedRunningTime="2026-11-03 17:44:56.45582383 +0000 UTC m=+6.800190210" Nov 03 17:44:57.225362 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:57.225315 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8b7gg" event={"ID":"fa56a595-a9ef-431d-b176-41cb1cf382d7","Type":"ContainerStarted","Data":"51ecde8a2a901562d4856c15521bcad69ca87c1d053a2bf9dc02d5f7b14422ee"} Nov 03 17:44:57.341741 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:57.341480 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8b7gg" podStartSLOduration=3.373624389 podStartE2EDuration="7.341455383s" totalImagesPullingTime="3.967830994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:57.341131776 +0000 UTC m=+7.685498178" watchObservedRunningTime="2026-11-03 17:44:57.341455383 +0000 UTC m=+7.685821760" Nov 03 17:44:57.467524 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:57.467448 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8b7gg" Nov 03 17:44:58.179545 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:58.179515 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:44:58.179708 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:58.179657 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs77w" podUID="0ba1ef12-3f5d-4449-8125-4a1921ec2990" Nov 03 17:44:58.179708 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:58.179693 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:44:58.179817 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:58.179765 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p8q8" podUID="d2c0ef79-ea05-4061-b4dd-31edac551213" Nov 03 17:44:58.179907 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:58.179818 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:44:58.179907 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:58.179887 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zpkh" podUID="c527a13f-da23-4228-a048-4af962c36340" Nov 03 17:44:58.234572 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:58.234525 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xk8dp" event={"ID":"c9107231-c3e3-4d76-81e6-b276b828c9bc","Type":"ContainerStarted","Data":"36e0e6f831f802df51a9d56053d63cad2c269fc5e9752561189b8c7f5a4131d0"} Nov 03 17:44:59.240856 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:59.240309 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" event={"ID":"815e56fd-a11f-4a53-8d85-bfc073f33ea8","Type":"ContainerStarted","Data":"109c3020f9afda8d5052889a7696e683046cb2a60c5385453d01948b2dca709c"} Nov 03 17:44:59.892475 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:59.892078 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs\") pod \"network-metrics-daemon-fs77w\" (UID: \"0ba1ef12-3f5d-4449-8125-4a1921ec2990\") " pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:44:59.892475 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:59.892130 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret\") pod \"global-pull-secret-syncer-8zpkh\" (UID: \"c527a13f-da23-4228-a048-4af962c36340\") " pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:44:59.892475 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:59.892289 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 17:44:59.892475 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:59.892342 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret podName:c527a13f-da23-4228-a048-4af962c36340 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:07.892330462 +0000 UTC m=+18.236696817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret") pod "global-pull-secret-syncer-8zpkh" (UID: "c527a13f-da23-4228-a048-4af962c36340") : object "kube-system"/"original-pull-secret" not registered Nov 03 17:44:59.892475 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:59.892417 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:44:59.892475 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:59.892447 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs podName:0ba1ef12-3f5d-4449-8125-4a1921ec2990 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:07.89243944 +0000 UTC m=+18.236805794 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs") pod "network-metrics-daemon-fs77w" (UID: "0ba1ef12-3f5d-4449-8125-4a1921ec2990") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:44:59.993451 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:44:59.993226 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48n9p\" (UniqueName: \"kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p\") pod \"network-check-target-8p8q8\" (UID: \"d2c0ef79-ea05-4061-b4dd-31edac551213\") " pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:44:59.993451 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:59.993342 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 17:44:59.993451 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:59.993355 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 17:44:59.993451 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:59.993363 2574 projected.go:196] Error preparing data for projected volume kube-api-access-48n9p for pod openshift-network-diagnostics/network-check-target-8p8q8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:44:59.993451 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:44:59.993423 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p podName:d2c0ef79-ea05-4061-b4dd-31edac551213 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:07.993406111 +0000 UTC m=+18.337772480 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-48n9p" (UniqueName: "kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p") pod "network-check-target-8p8q8" (UID: "d2c0ef79-ea05-4061-b4dd-31edac551213") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:45:00.188008 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:00.187977 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:00.188161 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:00.188135 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs77w" podUID="0ba1ef12-3f5d-4449-8125-4a1921ec2990" Nov 03 17:45:00.188239 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:00.188218 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:00.188350 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:00.188314 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p8q8" podUID="d2c0ef79-ea05-4061-b4dd-31edac551213" Nov 03 17:45:00.188801 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:00.188775 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:00.188936 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:00.188909 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zpkh" podUID="c527a13f-da23-4228-a048-4af962c36340" Nov 03 17:45:00.355876 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:00.355826 2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 17:45:01.190020 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:01.189913 2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T17:45:00.355850492Z","UUID":"7fe04736-e312-4563-a908-4c521db3e9e4","Handler":null,"Name":"","Endpoint":""} Nov 03 17:45:01.191551 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:01.191526 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 17:45:01.191656 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:01.191560 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 17:45:01.246434 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:01.245937 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4hhwj" event={"ID":"62507a50-2bff-4f45-82e7-aca45c032546","Type":"ContainerStarted","Data":"67d280691350aedaf9294c80af3049f040898132562464c30f3e889824623829"} Nov 03 17:45:01.249156 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:01.248520 2574 generic.go:356] "Generic (PLEG): container finished" podID="d3a9f32d-0e80-42ce-81ad-d4c08790fccb" containerID="3ced9032244d3922a3430c576f33b787eb8c9336538a8488e80bd7069b51734b" exitCode=0 Nov 03 17:45:01.249156 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:01.248835 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwttb" event={"ID":"d3a9f32d-0e80-42ce-81ad-d4c08790fccb","Type":"ContainerDied","Data":"3ced9032244d3922a3430c576f33b787eb8c9336538a8488e80bd7069b51734b"} Nov 03 17:45:01.250735 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:01.250706 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qjbks" event={"ID":"32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93","Type":"ContainerStarted","Data":"c5c28fd32b3e30be9666d63e21d68cd5661e265d291d7838260ebe9f182e7a63"} Nov 03 17:45:01.252884 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:01.252836 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" event={"ID":"815e56fd-a11f-4a53-8d85-bfc073f33ea8","Type":"ContainerStarted","Data":"2bc27facf0c82f315377488a563d29e8268f4b400c4abc67f4e9447faa102a9e"} Nov 03 17:45:01.341916 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:01.341343 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8b7gg" Nov 03 17:45:01.342301 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:01.342281 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8b7gg" Nov 03 17:45:01.365328 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:01.365270 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xk8dp" podStartSLOduration=6.477041172 podStartE2EDuration="11.365250406s" totalImagesPullingTime="4.888209234s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:58.351459341 +0000 UTC m=+8.695825721" watchObservedRunningTime="2026-11-03 17:45:01.365250406 +0000 UTC m=+11.709616785" Nov 03 17:45:01.484213 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:01.483970 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-4hhwj" podStartSLOduration=3.659879423 podStartE2EDuration="11.483954837s" totalImagesPullingTime="7.824075414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:01.3667052 +0000 UTC m=+11.711071556" watchObservedRunningTime="2026-11-03 17:45:01.483954837 +0000 UTC m=+11.828321213" Nov 03 17:45:01.723264 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:01.723210 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-qjbks" podStartSLOduration=3.890884709 podStartE2EDuration="11.723197496s" totalImagesPullingTime="7.832312787s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:01.600683236 +0000 UTC m=+11.945049613" watchObservedRunningTime="2026-11-03 17:45:01.723197496 +0000 UTC m=+12.067563873" Nov 03 17:45:02.179217 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:02.179185 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:02.179421 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:02.179251 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:02.179421 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:02.179252 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:02.179421 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:02.179350 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zpkh" podUID="c527a13f-da23-4228-a048-4af962c36340" Nov 03 17:45:02.179553 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:02.179470 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs77w" podUID="0ba1ef12-3f5d-4449-8125-4a1921ec2990" Nov 03 17:45:02.179595 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:02.179553 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p8q8" podUID="d2c0ef79-ea05-4061-b4dd-31edac551213" Nov 03 17:45:02.258289 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:02.257909 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-njt29" event={"ID":"dc524cbd-84c3-4255-a208-26fd55af457e","Type":"ContainerStarted","Data":"46aa69f2ceac59632cb805e23faf09b029727bbc99868ad228b12db070cc7566"} Nov 03 17:45:02.260745 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:02.260707 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" event={"ID":"815e56fd-a11f-4a53-8d85-bfc073f33ea8","Type":"ContainerStarted","Data":"ff75c33b4893ae5d183c14ead1f1e66f45d230652b28222a3c3390464626c050"} Nov 03 17:45:02.261454 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:02.261432 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8b7gg" Nov 03 17:45:02.374979 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:02.374921 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-njt29" podStartSLOduration=4.378229406 podStartE2EDuration="12.374900716s" totalImagesPullingTime="7.99667131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:02.37400902 +0000 UTC m=+12.718375397" watchObservedRunningTime="2026-11-03 17:45:02.374900716 +0000 UTC m=+12.719267095" Nov 03 17:45:02.608633 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:02.608591 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" podStartSLOduration=3.7472428410000003 podStartE2EDuration="12.608577917s" totalImagesPullingTime="8.861335076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:02.608066702 +0000 UTC m=+12.952433079" watchObservedRunningTime="2026-11-03 17:45:02.608577917 +0000 UTC m=+12.952944293" Nov 03 17:45:03.265013 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:03.264960 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" event={"ID":"0f7aa991-10ec-45ff-ac60-547adf0501a2","Type":"ContainerStarted","Data":"4cf90e85b552515c32cb6281b8bd957125ad6d725a0470791995b0f61ac74f82"} Nov 03 17:45:03.267493 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:03.267463 2574 generic.go:356] "Generic (PLEG): container finished" podID="d3a9f32d-0e80-42ce-81ad-d4c08790fccb" containerID="4114ff5cd7a69108d4b88c0578c3be346416f484adef1c25715d7c22558340f8" exitCode=0 Nov 03 17:45:03.267616 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:03.267516 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwttb" event={"ID":"d3a9f32d-0e80-42ce-81ad-d4c08790fccb","Type":"ContainerDied","Data":"4114ff5cd7a69108d4b88c0578c3be346416f484adef1c25715d7c22558340f8"} Nov 03 17:45:03.388312 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:03.388245 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" podStartSLOduration=3.380320558 podStartE2EDuration="13.388225408s" totalImagesPullingTime="10.00790485s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:03.387643617 +0000 UTC m=+13.732009995" watchObservedRunningTime="2026-11-03 17:45:03.388225408 +0000 UTC m=+13.732591786" Nov 03 17:45:04.180625 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:04.179797 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:04.180625 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:04.179928 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p8q8" podUID="d2c0ef79-ea05-4061-b4dd-31edac551213" Nov 03 17:45:04.180625 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:04.180335 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:04.180625 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:04.180459 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs77w" podUID="0ba1ef12-3f5d-4449-8125-4a1921ec2990" Nov 03 17:45:04.180625 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:04.180517 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:04.180625 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:04.180587 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zpkh" podUID="c527a13f-da23-4228-a048-4af962c36340" Nov 03 17:45:05.274882 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:05.274848 2574 generic.go:356] "Generic (PLEG): container finished" podID="d3a9f32d-0e80-42ce-81ad-d4c08790fccb" containerID="9a3105728ee2a2ba11b1b035a3c07d822ecac7ec085b41978efc015e96be8865" exitCode=0 Nov 03 17:45:05.275527 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:05.274896 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwttb" event={"ID":"d3a9f32d-0e80-42ce-81ad-d4c08790fccb","Type":"ContainerDied","Data":"9a3105728ee2a2ba11b1b035a3c07d822ecac7ec085b41978efc015e96be8865"} Nov 03 17:45:06.179724 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:06.179481 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:06.179873 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:06.179503 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:06.179873 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:06.179831 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p8q8" podUID="d2c0ef79-ea05-4061-b4dd-31edac551213" Nov 03 17:45:06.179991 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:06.179530 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:06.179991 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:06.179894 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs77w" podUID="0ba1ef12-3f5d-4449-8125-4a1921ec2990" Nov 03 17:45:06.179991 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:06.179978 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zpkh" podUID="c527a13f-da23-4228-a048-4af962c36340" Nov 03 17:45:07.046433 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:07.046386 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:07.281530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:07.281509 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 17:45:07.281842 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:07.281818 2574 generic.go:356] "Generic (PLEG): container finished" podID="6a74bba5-bc9c-4ae1-9754-8887c5bf23cc" containerID="340a940ba8b8964d841361a71a7d138c5ba910906e6863f9cfc28e6dce4b6469" exitCode=1 Nov 03 17:45:07.281927 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:07.281872 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" event={"ID":"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc","Type":"ContainerStarted","Data":"94d5f4ce3034a033cf8008a79a546df21a3346400cda66d48cd27609b01b0f27"} Nov 03 17:45:07.281990 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:07.281953 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" event={"ID":"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc","Type":"ContainerDied","Data":"340a940ba8b8964d841361a71a7d138c5ba910906e6863f9cfc28e6dce4b6469"} Nov 03 17:45:07.282114 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:07.281991 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" event={"ID":"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc","Type":"ContainerStarted","Data":"d7cc82e7c317a34d62ac79fb58b89c4b2c3d774f09a69f24107792d2f91b8fe5"} Nov 03 17:45:07.289136 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:07.289104 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:07.543195 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:07.543037 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:07.955012 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:07.954933 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs\") pod \"network-metrics-daemon-fs77w\" (UID: \"0ba1ef12-3f5d-4449-8125-4a1921ec2990\") " pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:07.955012 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:07.954966 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret\") pod \"global-pull-secret-syncer-8zpkh\" (UID: \"c527a13f-da23-4228-a048-4af962c36340\") " pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:07.955199 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:07.955065 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 17:45:07.955199 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:07.955108 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret podName:c527a13f-da23-4228-a048-4af962c36340 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:23.955096816 +0000 UTC m=+34.299463171 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret") pod "global-pull-secret-syncer-8zpkh" (UID: "c527a13f-da23-4228-a048-4af962c36340") : object "kube-system"/"original-pull-secret" not registered Nov 03 17:45:07.955199 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:07.955065 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:45:07.955199 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:07.955140 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs podName:0ba1ef12-3f5d-4449-8125-4a1921ec2990 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:23.95513045 +0000 UTC m=+34.299496805 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs") pod "network-metrics-daemon-fs77w" (UID: "0ba1ef12-3f5d-4449-8125-4a1921ec2990") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:45:08.011560 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:08.011530 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:08.056216 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:08.056178 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48n9p\" (UniqueName: \"kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p\") pod \"network-check-target-8p8q8\" (UID: \"d2c0ef79-ea05-4061-b4dd-31edac551213\") " pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:08.056542 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:08.056325 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 17:45:08.056542 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:08.056342 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 17:45:08.056542 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:08.056351 2574 projected.go:196] Error preparing data for projected volume kube-api-access-48n9p for pod openshift-network-diagnostics/network-check-target-8p8q8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:45:08.056542 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:08.056416 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p podName:d2c0ef79-ea05-4061-b4dd-31edac551213 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:24.056384563 +0000 UTC m=+34.400750933 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-48n9p" (UniqueName: "kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p") pod "network-check-target-8p8q8" (UID: "d2c0ef79-ea05-4061-b4dd-31edac551213") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:45:08.179072 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:08.179036 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:08.179255 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:08.179081 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:08.179255 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:08.179041 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:08.179255 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:08.179160 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zpkh" podUID="c527a13f-da23-4228-a048-4af962c36340" Nov 03 17:45:08.179443 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:08.179251 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p8q8" podUID="d2c0ef79-ea05-4061-b4dd-31edac551213" Nov 03 17:45:08.179443 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:08.179329 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs77w" podUID="0ba1ef12-3f5d-4449-8125-4a1921ec2990" Nov 03 17:45:08.286515 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:08.286428 2574 generic.go:356] "Generic (PLEG): container finished" podID="d3a9f32d-0e80-42ce-81ad-d4c08790fccb" containerID="5ae034fee90524865e2694fe0c37da036b6443201ee3379f601cb1928263a4fb" exitCode=0 Nov 03 17:45:08.286669 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:08.286511 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwttb" event={"ID":"d3a9f32d-0e80-42ce-81ad-d4c08790fccb","Type":"ContainerDied","Data":"5ae034fee90524865e2694fe0c37da036b6443201ee3379f601cb1928263a4fb"} Nov 03 17:45:08.289343 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:08.289329 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 17:45:08.289644 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:08.289624 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" event={"ID":"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc","Type":"ContainerStarted","Data":"c8e7f822480b219c8f8c26b42ad31f1433c5558633a2553e075b7adcf4adb316"} Nov 03 17:45:08.289720 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:08.289679 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" event={"ID":"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc","Type":"ContainerStarted","Data":"1401331a666c3e1503755519cb3996175b7773e963f2e656cb8aa6177cc934b8"} Nov 03 17:45:08.289720 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:08.289710 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" event={"ID":"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc","Type":"ContainerStarted","Data":"057391eb33a4b254b7e4200dab10a77f5c65a721e594d176e464451a8ad99f49"} Nov 03 17:45:09.295293 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:09.295261 2574 generic.go:356] "Generic (PLEG): container finished" podID="d3a9f32d-0e80-42ce-81ad-d4c08790fccb" containerID="f4e07f7d3464bcdbbd319d25d4fed6106f0685beca6c3cff55cfdaf96582e1c2" exitCode=0 Nov 03 17:45:09.295764 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:09.295307 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwttb" event={"ID":"d3a9f32d-0e80-42ce-81ad-d4c08790fccb","Type":"ContainerDied","Data":"f4e07f7d3464bcdbbd319d25d4fed6106f0685beca6c3cff55cfdaf96582e1c2"} Nov 03 17:45:10.180598 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:10.180563 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:10.180778 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:10.180681 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p8q8" podUID="d2c0ef79-ea05-4061-b4dd-31edac551213" Nov 03 17:45:10.180778 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:10.180726 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:10.180778 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:10.180761 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:10.180889 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:10.180821 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zpkh" podUID="c527a13f-da23-4228-a048-4af962c36340" Nov 03 17:45:10.180927 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:10.180896 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs77w" podUID="0ba1ef12-3f5d-4449-8125-4a1921ec2990" Nov 03 17:45:10.300245 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:10.300218 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 17:45:10.300780 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:10.300628 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" event={"ID":"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc","Type":"ContainerStarted","Data":"4d95620cc9daeea7c4ba953df8b4c1dd980ece0a6485726626815d974e6aba56"} Nov 03 17:45:10.303558 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:10.303537 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwttb" event={"ID":"d3a9f32d-0e80-42ce-81ad-d4c08790fccb","Type":"ContainerStarted","Data":"7067eccc24ff5645d203540b0fe3cbc2cc235003a0b329215421dbd0ca36ddbc"} Nov 03 17:45:10.467284 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:10.467178 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fwttb" podStartSLOduration=8.418926798 podStartE2EDuration="20.467163969s" totalImagesPullingTime="12.048237171s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:10.466858533 +0000 UTC m=+20.811224911" watchObservedRunningTime="2026-11-03 17:45:10.467163969 +0000 UTC m=+20.811530345" Nov 03 17:45:11.694694 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:11.694658 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:11.950939 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:11.950861 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:12.158170 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:12.158141 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:12.180079 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:12.180050 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:12.180243 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:12.180188 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:12.180243 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:12.180205 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs77w" podUID="0ba1ef12-3f5d-4449-8125-4a1921ec2990" Nov 03 17:45:12.180364 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:12.180306 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p8q8" podUID="d2c0ef79-ea05-4061-b4dd-31edac551213" Nov 03 17:45:12.180364 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:12.180349 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:12.180486 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:12.180424 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zpkh" podUID="c527a13f-da23-4228-a048-4af962c36340" Nov 03 17:45:12.310728 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:12.310658 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 17:45:12.311066 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:12.311039 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" event={"ID":"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc","Type":"ContainerStarted","Data":"37ae7246bb423f8f26a7e8e1229b221cd48ca4819bd346ad4f561f2d344c6a11"} Nov 03 17:45:12.311344 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:12.311323 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:45:12.311496 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:12.311383 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:45:12.311496 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:12.311485 2574 scope.go:122] "RemoveContainer" containerID="340a940ba8b8964d841361a71a7d138c5ba910906e6863f9cfc28e6dce4b6469" Nov 03 17:45:12.326898 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:12.326767 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:45:12.369816 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:12.369795 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:13.311044 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:13.311005 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:13.318096 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:13.318076 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 17:45:13.318500 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:13.318480 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" event={"ID":"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc","Type":"ContainerStarted","Data":"7b64974dacbb30e469e98364f15d380e4eed23c9cb1b17ebdb70c8af8e13affb"} Nov 03 17:45:13.318753 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:13.318730 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:45:13.333485 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:13.333464 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:45:13.503124 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:13.503078 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" podStartSLOduration=9.034456098 podStartE2EDuration="23.503065814s" totalImagesPullingTime="14.468609716s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:13.502604791 +0000 UTC m=+23.846971169" watchObservedRunningTime="2026-11-03 17:45:13.503065814 +0000 UTC m=+23.847432211" Nov 03 17:45:13.518266 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:13.518236 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:13.800675 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:13.800596 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:14.187095 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:14.187028 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:14.187095 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:14.187050 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:14.187315 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:14.187207 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:14.187315 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:14.187205 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zpkh" podUID="c527a13f-da23-4228-a048-4af962c36340" Nov 03 17:45:14.187434 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:14.187332 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs77w" podUID="0ba1ef12-3f5d-4449-8125-4a1921ec2990" Nov 03 17:45:14.187496 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:14.187437 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p8q8" podUID="d2c0ef79-ea05-4061-b4dd-31edac551213" Nov 03 17:45:14.294326 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:14.294289 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:14.384829 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:14.384799 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8zpkh"] Nov 03 17:45:14.385230 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:14.384890 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:14.385230 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:14.384965 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zpkh" podUID="c527a13f-da23-4228-a048-4af962c36340" Nov 03 17:45:14.389425 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:14.389382 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8p8q8"] Nov 03 17:45:14.389557 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:14.389484 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:14.389615 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:14.389556 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p8q8" podUID="d2c0ef79-ea05-4061-b4dd-31edac551213" Nov 03 17:45:14.389985 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:14.389964 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fs77w"] Nov 03 17:45:14.390077 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:14.390063 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:14.390183 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:14.390166 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs77w" podUID="0ba1ef12-3f5d-4449-8125-4a1921ec2990" Nov 03 17:45:15.219704 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:15.219668 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:15.744737 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:15.744705 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:15.977319 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:15.977245 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:16.070188 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:16.070166 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qjbks_32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93/dns-node-resolver/0.log" Nov 03 17:45:16.179847 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:16.179821 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:16.180020 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:16.179880 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:16.180020 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:16.179892 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:16.180020 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:16.179976 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zpkh" podUID="c527a13f-da23-4228-a048-4af962c36340" Nov 03 17:45:16.180174 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:16.180070 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs77w" podUID="0ba1ef12-3f5d-4449-8125-4a1921ec2990" Nov 03 17:45:16.180174 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:16.180134 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p8q8" podUID="d2c0ef79-ea05-4061-b4dd-31edac551213" Nov 03 17:45:16.181909 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:16.181885 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:16.393444 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:16.393407 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:16.970741 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:16.970718 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xk8dp_c9107231-c3e3-4d76-81e6-b276b828c9bc/node-ca/0.log" Nov 03 17:45:16.972923 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:16.972891 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:17.237510 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:17.237433 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:17.481995 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:17.481964 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:17.695613 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:17.695537 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:17.901612 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:17.901557 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:18.182726 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:18.182657 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:18.183094 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:18.182750 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zpkh" podUID="c527a13f-da23-4228-a048-4af962c36340" Nov 03 17:45:18.183094 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:18.182824 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:18.183094 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:18.182909 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p8q8" podUID="d2c0ef79-ea05-4061-b4dd-31edac551213" Nov 03 17:45:18.183094 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:18.182946 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:18.183094 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:18.182999 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs77w" podUID="0ba1ef12-3f5d-4449-8125-4a1921ec2990" Nov 03 17:45:18.373969 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:18.373938 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:18.973960 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:18.973868 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:19.191485 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:19.191453 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:19.577656 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:19.577620 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:19.811337 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:19.811302 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:20.180284 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:20.180245 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:20.180494 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:20.180337 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:20.180494 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:20.180473 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zpkh" podUID="c527a13f-da23-4228-a048-4af962c36340" Nov 03 17:45:20.180612 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:20.180544 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:20.180612 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:20.180543 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p8q8" podUID="d2c0ef79-ea05-4061-b4dd-31edac551213" Nov 03 17:45:20.180705 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:20.180676 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs77w" podUID="0ba1ef12-3f5d-4449-8125-4a1921ec2990" Nov 03 17:45:20.236984 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:20.236915 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:20.505791 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:20.505695 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:20.986778 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:20.986687 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:21.247498 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:21.247414 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:21.571832 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:21.571751 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:21.781586 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:21.781551 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:21.979885 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:21.979806 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:22.180001 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:22.179970 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:22.180001 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:22.180000 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:22.180237 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:22.180076 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zpkh" podUID="c527a13f-da23-4228-a048-4af962c36340" Nov 03 17:45:22.180237 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:22.180130 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:22.180237 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:22.180200 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p8q8" podUID="d2c0ef79-ea05-4061-b4dd-31edac551213" Nov 03 17:45:22.180367 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:22.180268 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs77w" podUID="0ba1ef12-3f5d-4449-8125-4a1921ec2990" Nov 03 17:45:22.375967 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:22.375889 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:22.994731 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:22.994703 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:23.251597 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:23.251509 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:23.459633 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:23.459578 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:23.665852 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:23.665821 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:23.870306 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:23.870270 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:23.979359 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:23.979326 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs\") pod \"network-metrics-daemon-fs77w\" (UID: \"0ba1ef12-3f5d-4449-8125-4a1921ec2990\") " pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:23.979508 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:23.979366 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret\") pod \"global-pull-secret-syncer-8zpkh\" (UID: \"c527a13f-da23-4228-a048-4af962c36340\") " pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:23.979508 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:23.979491 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:45:23.979599 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:23.979509 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 17:45:23.979599 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:23.979553 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs podName:0ba1ef12-3f5d-4449-8125-4a1921ec2990 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:55.979538432 +0000 UTC m=+66.323904791 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs") pod "network-metrics-daemon-fs77w" (UID: "0ba1ef12-3f5d-4449-8125-4a1921ec2990") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:45:23.979599 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:23.979567 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret podName:c527a13f-da23-4228-a048-4af962c36340 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:55.979560534 +0000 UTC m=+66.323926889 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret") pod "global-pull-secret-syncer-8zpkh" (UID: "c527a13f-da23-4228-a048-4af962c36340") : object "kube-system"/"original-pull-secret" not registered Nov 03 17:45:24.080163 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:24.080131 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48n9p\" (UniqueName: \"kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p\") pod \"network-check-target-8p8q8\" (UID: \"d2c0ef79-ea05-4061-b4dd-31edac551213\") " pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:24.080313 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:24.080249 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 17:45:24.080313 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:24.080263 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 17:45:24.080313 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:24.080272 2574 projected.go:196] Error preparing data for projected volume kube-api-access-48n9p for pod openshift-network-diagnostics/network-check-target-8p8q8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:45:24.080436 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:24.080316 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p podName:d2c0ef79-ea05-4061-b4dd-31edac551213 nodeName:}" failed. No retries permitted until 2026-11-03 17:45:56.080304069 +0000 UTC m=+66.424670423 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-48n9p" (UniqueName: "kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p") pod "network-check-target-8p8q8" (UID: "d2c0ef79-ea05-4061-b4dd-31edac551213") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:45:24.179888 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:24.179857 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:24.180026 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:24.179858 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:24.180026 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:24.179962 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs77w" podUID="0ba1ef12-3f5d-4449-8125-4a1921ec2990" Nov 03 17:45:24.180026 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:24.179987 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:24.180136 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:24.180029 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p8q8" podUID="d2c0ef79-ea05-4061-b4dd-31edac551213" Nov 03 17:45:24.180136 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:24.180085 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zpkh" podUID="c527a13f-da23-4228-a048-4af962c36340" Nov 03 17:45:24.283092 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:24.283003 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:24.488684 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:24.488649 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:24.990419 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:24.990372 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:25.191054 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:25.191021 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:25.390022 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:25.389947 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:25.599651 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:25.599617 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:25.797631 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:25.797548 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:26.179119 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:26.179045 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:26.179276 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:26.179055 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:26.179276 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:26.179163 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:26.179276 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:26.179233 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs77w" podUID="0ba1ef12-3f5d-4449-8125-4a1921ec2990" Nov 03 17:45:26.179276 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:26.179140 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p8q8" podUID="d2c0ef79-ea05-4061-b4dd-31edac551213" Nov 03 17:45:26.179421 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:26.179343 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zpkh" podUID="c527a13f-da23-4228-a048-4af962c36340" Nov 03 17:45:26.377857 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:26.377822 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:26.634352 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:26.634260 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:26.848817 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:26.848770 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:28.179874 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:28.179844 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:28.179874 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:28.179858 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:28.180288 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:28.179940 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs77w" podUID="0ba1ef12-3f5d-4449-8125-4a1921ec2990" Nov 03 17:45:28.180288 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:28.179987 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:28.180288 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:28.180060 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zpkh" podUID="c527a13f-da23-4228-a048-4af962c36340" Nov 03 17:45:28.180288 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:28.180115 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p8q8" podUID="d2c0ef79-ea05-4061-b4dd-31edac551213" Nov 03 17:45:30.179963 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:30.179929 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:30.180354 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:30.180020 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs77w" podUID="0ba1ef12-3f5d-4449-8125-4a1921ec2990" Nov 03 17:45:30.180354 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:30.180116 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:30.180354 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:30.180233 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:30.180354 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:30.180238 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zpkh" podUID="c527a13f-da23-4228-a048-4af962c36340" Nov 03 17:45:30.180354 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:30.180310 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p8q8" podUID="d2c0ef79-ea05-4061-b4dd-31edac551213" Nov 03 17:45:32.031834 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:32.031747 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:32.179127 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:32.179091 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:32.179302 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:32.179091 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:32.179302 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:32.179196 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p8q8" podUID="d2c0ef79-ea05-4061-b4dd-31edac551213" Nov 03 17:45:32.179302 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:32.179226 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:32.179302 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:32.179290 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zpkh" podUID="c527a13f-da23-4228-a048-4af962c36340" Nov 03 17:45:32.179559 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:32.179380 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs77w" podUID="0ba1ef12-3f5d-4449-8125-4a1921ec2990" Nov 03 17:45:34.179698 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:34.179666 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:34.179698 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:34.179677 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:34.179698 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:34.179700 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:34.180216 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:34.179772 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8p8q8" podUID="d2c0ef79-ea05-4061-b4dd-31edac551213" Nov 03 17:45:34.180216 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:34.179848 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zpkh" podUID="c527a13f-da23-4228-a048-4af962c36340" Nov 03 17:45:34.180216 ip-10-0-137-164 kubenswrapper[2574]: E1103 17:45:34.179952 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs77w" podUID="0ba1ef12-3f5d-4449-8125-4a1921ec2990" Nov 03 17:45:34.371541 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:34.371506 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:34.655846 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:34.655756 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:34.994689 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:34.994603 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:35.200278 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.200249 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:35.354949 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.354919 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeReady" Nov 03 17:45:35.355086 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.355038 2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 17:45:35.511530 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.511503 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fkhbf"] Nov 03 17:45:35.513591 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.513577 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fkhbf" Nov 03 17:45:35.525151 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.525126 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rbt65"] Nov 03 17:45:35.528108 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.528088 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-khc2s"] Nov 03 17:45:35.528216 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.528198 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:45:35.530260 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.530240 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fkhbf"] Nov 03 17:45:35.530338 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.530276 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rbt65"] Nov 03 17:45:35.530338 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.530324 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-khc2s" Nov 03 17:45:35.566094 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.566070 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189-config-volume\") pod \"dns-default-khc2s\" (UID: \"4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189\") " pod="openshift-dns/dns-default-khc2s" Nov 03 17:45:35.566192 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.566096 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18c90190-94ae-4270-b735-a46d3aaa3f42-crio-socket\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:45:35.566192 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.566113 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18c90190-94ae-4270-b735-a46d3aaa3f42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:45:35.566192 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.566131 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmhkc\" (UniqueName: \"kubernetes.io/projected/40fbf434-33db-4779-acf2-a148d4cb8c60-kube-api-access-hmhkc\") pod \"ingress-canary-fkhbf\" (UID: \"40fbf434-33db-4779-acf2-a148d4cb8c60\") " pod="openshift-ingress-canary/ingress-canary-fkhbf" Nov 03 17:45:35.566360 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.566196 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40fbf434-33db-4779-acf2-a148d4cb8c60-cert\") pod \"ingress-canary-fkhbf\" (UID: \"40fbf434-33db-4779-acf2-a148d4cb8c60\") " pod="openshift-ingress-canary/ingress-canary-fkhbf" Nov 03 17:45:35.566360 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.566302 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18c90190-94ae-4270-b735-a46d3aaa3f42-data-volume\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:45:35.566360 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.566344 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189-tmp-dir\") pod \"dns-default-khc2s\" (UID: \"4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189\") " pod="openshift-dns/dns-default-khc2s" Nov 03 17:45:35.566491 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.566372 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189-metrics-tls\") pod \"dns-default-khc2s\" (UID: \"4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189\") " pod="openshift-dns/dns-default-khc2s" Nov 03 17:45:35.566491 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.566437 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljxz9\" (UniqueName: \"kubernetes.io/projected/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189-kube-api-access-ljxz9\") pod \"dns-default-khc2s\" (UID: \"4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189\") " pod="openshift-dns/dns-default-khc2s" Nov 03 17:45:35.566491 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.566476 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5flr\" (UniqueName: \"kubernetes.io/projected/18c90190-94ae-4270-b735-a46d3aaa3f42-kube-api-access-d5flr\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:45:35.566626 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.566540 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18c90190-94ae-4270-b735-a46d3aaa3f42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:45:35.579654 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.579632 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-khc2s"] Nov 03 17:45:35.614363 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.614303 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 17:45:35.614507 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.614483 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 17:45:35.614507 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.614500 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 17:45:35.614666 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.614569 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xfh9g\"" Nov 03 17:45:35.614666 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.614619 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 17:45:35.614666 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.614650 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 17:45:35.614791 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.614687 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 17:45:35.614791 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.614735 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 17:45:35.614865 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.614797 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 17:45:35.614865 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.614809 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 17:45:35.614865 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.614846 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nvxmw\"" Nov 03 17:45:35.614987 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.614928 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pvbgk\"" Nov 03 17:45:35.656521 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.656498 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:35.667448 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.667425 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5flr\" (UniqueName: \"kubernetes.io/projected/18c90190-94ae-4270-b735-a46d3aaa3f42-kube-api-access-d5flr\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:45:35.667538 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.667459 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18c90190-94ae-4270-b735-a46d3aaa3f42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:45:35.667538 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.667518 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189-config-volume\") pod \"dns-default-khc2s\" (UID: \"4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189\") " pod="openshift-dns/dns-default-khc2s" Nov 03 17:45:35.667645 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.667545 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18c90190-94ae-4270-b735-a46d3aaa3f42-crio-socket\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:45:35.667645 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.667568 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18c90190-94ae-4270-b735-a46d3aaa3f42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:45:35.667645 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.667596 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmhkc\" (UniqueName: \"kubernetes.io/projected/40fbf434-33db-4779-acf2-a148d4cb8c60-kube-api-access-hmhkc\") pod \"ingress-canary-fkhbf\" (UID: \"40fbf434-33db-4779-acf2-a148d4cb8c60\") " pod="openshift-ingress-canary/ingress-canary-fkhbf" Nov 03 17:45:35.667645 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.667622 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40fbf434-33db-4779-acf2-a148d4cb8c60-cert\") pod \"ingress-canary-fkhbf\" (UID: \"40fbf434-33db-4779-acf2-a148d4cb8c60\") " pod="openshift-ingress-canary/ingress-canary-fkhbf" Nov 03 17:45:35.667839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.667684 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18c90190-94ae-4270-b735-a46d3aaa3f42-data-volume\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:45:35.667839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.667718 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189-tmp-dir\") pod \"dns-default-khc2s\" (UID: \"4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189\") " pod="openshift-dns/dns-default-khc2s" Nov 03 17:45:35.667839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.667743 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189-metrics-tls\") pod \"dns-default-khc2s\" (UID: \"4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189\") " pod="openshift-dns/dns-default-khc2s" Nov 03 17:45:35.667839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.667754 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18c90190-94ae-4270-b735-a46d3aaa3f42-crio-socket\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:45:35.667839 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.667767 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljxz9\" (UniqueName: \"kubernetes.io/projected/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189-kube-api-access-ljxz9\") pod \"dns-default-khc2s\" (UID: \"4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189\") " pod="openshift-dns/dns-default-khc2s" Nov 03 17:45:35.668112 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.668090 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189-tmp-dir\") pod \"dns-default-khc2s\" (UID: \"4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189\") " pod="openshift-dns/dns-default-khc2s" Nov 03 17:45:35.668242 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.668214 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18c90190-94ae-4270-b735-a46d3aaa3f42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:45:35.668315 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.668272 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18c90190-94ae-4270-b735-a46d3aaa3f42-data-volume\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:45:35.668357 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.668320 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189-config-volume\") pod \"dns-default-khc2s\" (UID: \"4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189\") " pod="openshift-dns/dns-default-khc2s" Nov 03 17:45:35.671210 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.671184 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189-metrics-tls\") pod \"dns-default-khc2s\" (UID: \"4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189\") " pod="openshift-dns/dns-default-khc2s" Nov 03 17:45:35.671288 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.671186 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18c90190-94ae-4270-b735-a46d3aaa3f42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:45:35.671327 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.671298 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40fbf434-33db-4779-acf2-a148d4cb8c60-cert\") pod \"ingress-canary-fkhbf\" (UID: \"40fbf434-33db-4779-acf2-a148d4cb8c60\") " pod="openshift-ingress-canary/ingress-canary-fkhbf" Nov 03 17:45:35.728684 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.728663 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljxz9\" (UniqueName: \"kubernetes.io/projected/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189-kube-api-access-ljxz9\") pod \"dns-default-khc2s\" (UID: \"4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189\") " pod="openshift-dns/dns-default-khc2s" Nov 03 17:45:35.728790 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.728714 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5flr\" (UniqueName: \"kubernetes.io/projected/18c90190-94ae-4270-b735-a46d3aaa3f42-kube-api-access-d5flr\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:45:35.728927 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.728911 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmhkc\" (UniqueName: \"kubernetes.io/projected/40fbf434-33db-4779-acf2-a148d4cb8c60-kube-api-access-hmhkc\") pod \"ingress-canary-fkhbf\" (UID: \"40fbf434-33db-4779-acf2-a148d4cb8c60\") " pod="openshift-ingress-canary/ingress-canary-fkhbf" Nov 03 17:45:35.829831 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.829792 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fkhbf" Nov 03 17:45:35.841075 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.841052 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:45:35.844222 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:35.844201 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-khc2s" Nov 03 17:45:36.179978 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:36.179892 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:36.179978 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:36.179914 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:36.180168 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:36.179997 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:36.233591 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:36.233569 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 17:45:36.233933 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:36.233608 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sq9vz\"" Nov 03 17:45:36.233933 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:36.233626 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 17:45:36.233933 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:36.233638 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 17:45:36.233933 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:36.233665 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 17:45:36.233933 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:36.233792 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f85t9\"" Nov 03 17:45:36.233933 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:36.233839 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6lzh9\"" Nov 03 17:45:36.276796 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:36.276506 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fkhbf"] Nov 03 17:45:36.279706 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:36.279684 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-khc2s"] Nov 03 17:45:36.280193 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:36.280173 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rbt65"] Nov 03 17:45:36.282144 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:45:36.282122 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40fbf434_33db_4779_acf2_a148d4cb8c60.slice/crio-e9773a1686de25e15f4703384ffceb3aa5599c28b1cb8d222263d28cbc30feed WatchSource:0}: Error finding container e9773a1686de25e15f4703384ffceb3aa5599c28b1cb8d222263d28cbc30feed: Status 404 returned error can't find the container with id e9773a1686de25e15f4703384ffceb3aa5599c28b1cb8d222263d28cbc30feed Nov 03 17:45:36.283009 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:45:36.282987 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd0f35f_4fe3_4ab9_b92f_ca8240e8a189.slice/crio-7eaa7527d445149c20c06e95d2e090058c2cb7f5f069061435fe92fc52a81e16 WatchSource:0}: Error finding container 7eaa7527d445149c20c06e95d2e090058c2cb7f5f069061435fe92fc52a81e16: Status 404 returned error can't find the container with id 7eaa7527d445149c20c06e95d2e090058c2cb7f5f069061435fe92fc52a81e16 Nov 03 17:45:36.283946 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:45:36.283929 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c90190_94ae_4270_b735_a46d3aaa3f42.slice/crio-b85290cd780a3d66a3e633e6c560f1ff0ccdb7ee32ac59f151f6cb529ee08235 WatchSource:0}: Error finding container b85290cd780a3d66a3e633e6c560f1ff0ccdb7ee32ac59f151f6cb529ee08235: Status 404 returned error can't find the container with id b85290cd780a3d66a3e633e6c560f1ff0ccdb7ee32ac59f151f6cb529ee08235 Nov 03 17:45:36.366259 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:36.366233 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rbt65" event={"ID":"18c90190-94ae-4270-b735-a46d3aaa3f42","Type":"ContainerStarted","Data":"e3db6d61341af0aeb8c5640b57e4cc3ab42066d890ffcfe467cf8993ea2bdf7d"} Nov 03 17:45:36.366358 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:36.366287 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rbt65" event={"ID":"18c90190-94ae-4270-b735-a46d3aaa3f42","Type":"ContainerStarted","Data":"b85290cd780a3d66a3e633e6c560f1ff0ccdb7ee32ac59f151f6cb529ee08235"} Nov 03 17:45:36.367449 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:36.367418 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-khc2s" event={"ID":"4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189","Type":"ContainerStarted","Data":"7eaa7527d445149c20c06e95d2e090058c2cb7f5f069061435fe92fc52a81e16"} Nov 03 17:45:36.368546 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:36.368519 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fkhbf" event={"ID":"40fbf434-33db-4779-acf2-a148d4cb8c60","Type":"ContainerStarted","Data":"e9773a1686de25e15f4703384ffceb3aa5599c28b1cb8d222263d28cbc30feed"} Nov 03 17:45:36.846824 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:36.846783 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:37.050539 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:37.050504 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:37.575023 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:37.574983 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:37.784885 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:37.784838 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:37.999323 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:37.999243 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:38.209046 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:38.209014 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:38.377966 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:38.377927 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rbt65" event={"ID":"18c90190-94ae-4270-b735-a46d3aaa3f42","Type":"ContainerStarted","Data":"d09c2a836844e50342ce6bd1916210ae06104aec9fe8e749f21252b99ee37c65"} Nov 03 17:45:40.384941 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:40.384905 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rbt65" event={"ID":"18c90190-94ae-4270-b735-a46d3aaa3f42","Type":"ContainerStarted","Data":"4da17b5ceaf1803d9a01d63966b315ce2f40744a924bcb15338d387df28363db"} Nov 03 17:45:40.386758 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:40.386736 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-khc2s" event={"ID":"4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189","Type":"ContainerStarted","Data":"01072032bbbf87cdab72292ced020096decd96d2fea2dcadde4b5324f0213b31"} Nov 03 17:45:40.386882 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:40.386775 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-khc2s" event={"ID":"4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189","Type":"ContainerStarted","Data":"33e76dd9844ec99eb634958ed5bf1d186c2bb0b529a6d61e85da9a4cff8c76f9"} Nov 03 17:45:40.386882 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:40.386796 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-khc2s" Nov 03 17:45:40.388152 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:40.388133 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fkhbf" event={"ID":"40fbf434-33db-4779-acf2-a148d4cb8c60","Type":"ContainerStarted","Data":"75cd9da4d1e43cc3cb195c1f6bf117d7e34d3e612186ab08f8f2eef2c0b09384"} Nov 03 17:45:40.653369 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:40.653278 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:40.907738 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:40.907648 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:41.481617 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:41.481565 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fkhbf" podStartSLOduration=3.394333385 podStartE2EDuration="6.481551963s" totalImagesPullingTime="3.087218578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:41.480932883 +0000 UTC m=+51.825299257" watchObservedRunningTime="2026-11-03 17:45:41.481551963 +0000 UTC m=+51.825918339" Nov 03 17:45:41.600912 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:41.600854 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rbt65" podStartSLOduration=3.350807729 podStartE2EDuration="6.600839445s" totalImagesPullingTime="3.250031716s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:45:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:41.599845307 +0000 UTC m=+51.944211683" watchObservedRunningTime="2026-11-03 17:45:41.600839445 +0000 UTC m=+51.945205823" Nov 03 17:45:41.718464 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:41.718385 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-khc2s" podStartSLOduration=3.100457881 podStartE2EDuration="6.718370768s" totalImagesPullingTime="3.617912887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:41.717830209 +0000 UTC m=+52.062196586" watchObservedRunningTime="2026-11-03 17:45:41.718370768 +0000 UTC m=+52.062737145" Nov 03 17:45:43.292772 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:43.292730 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:43.849034 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:43.848997 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:44.051304 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.051265 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:44.394690 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.394652 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r"] Nov 03 17:45:44.397217 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.397194 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" Nov 03 17:45:44.433788 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.433743 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78a8038b-489c-44f5-827f-d4aaebca42f8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-c5q4r\" (UID: \"78a8038b-489c-44f5-827f-d4aaebca42f8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" Nov 03 17:45:44.433921 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.433799 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78a8038b-489c-44f5-827f-d4aaebca42f8-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-c5q4r\" (UID: \"78a8038b-489c-44f5-827f-d4aaebca42f8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" Nov 03 17:45:44.433921 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.433865 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sm4t\" (UniqueName: \"kubernetes.io/projected/78a8038b-489c-44f5-827f-d4aaebca42f8-kube-api-access-9sm4t\") pod \"openshift-state-metrics-7bd7489957-c5q4r\" (UID: \"78a8038b-489c-44f5-827f-d4aaebca42f8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" Nov 03 17:45:44.434002 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.433936 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/78a8038b-489c-44f5-827f-d4aaebca42f8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-c5q4r\" (UID: \"78a8038b-489c-44f5-827f-d4aaebca42f8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" Nov 03 17:45:44.450046 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.450024 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 17:45:44.450360 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.450341 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-k2vg5\"" Nov 03 17:45:44.500459 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.500428 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 17:45:44.500459 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.500433 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 17:45:44.501037 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.501015 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 17:45:44.501082 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.501034 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 17:45:44.501921 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.501904 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r"] Nov 03 17:45:44.502510 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.502494 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc"] Nov 03 17:45:44.505173 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.505155 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:45:44.535150 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.535124 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83572d76-31ae-42e0-b7d4-acc08df84a9c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:45:44.535261 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.535173 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/78a8038b-489c-44f5-827f-d4aaebca42f8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-c5q4r\" (UID: \"78a8038b-489c-44f5-827f-d4aaebca42f8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" Nov 03 17:45:44.535261 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.535201 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78a8038b-489c-44f5-827f-d4aaebca42f8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-c5q4r\" (UID: \"78a8038b-489c-44f5-827f-d4aaebca42f8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" Nov 03 17:45:44.535341 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.535257 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78a8038b-489c-44f5-827f-d4aaebca42f8-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-c5q4r\" (UID: \"78a8038b-489c-44f5-827f-d4aaebca42f8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" Nov 03 17:45:44.535341 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.535283 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/83572d76-31ae-42e0-b7d4-acc08df84a9c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:45:44.535341 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.535304 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sm4t\" (UniqueName: \"kubernetes.io/projected/78a8038b-489c-44f5-827f-d4aaebca42f8-kube-api-access-9sm4t\") pod \"openshift-state-metrics-7bd7489957-c5q4r\" (UID: \"78a8038b-489c-44f5-827f-d4aaebca42f8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" Nov 03 17:45:44.535341 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.535334 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83572d76-31ae-42e0-b7d4-acc08df84a9c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:45:44.535534 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.535363 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/83572d76-31ae-42e0-b7d4-acc08df84a9c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:45:44.535534 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.535390 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnv9j\" (UniqueName: \"kubernetes.io/projected/83572d76-31ae-42e0-b7d4-acc08df84a9c-kube-api-access-mnv9j\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:45:44.535637 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.535580 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/83572d76-31ae-42e0-b7d4-acc08df84a9c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:45:44.535925 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.535895 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78a8038b-489c-44f5-827f-d4aaebca42f8-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-c5q4r\" (UID: \"78a8038b-489c-44f5-827f-d4aaebca42f8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" Nov 03 17:45:44.537658 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.537635 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78a8038b-489c-44f5-827f-d4aaebca42f8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-c5q4r\" (UID: \"78a8038b-489c-44f5-827f-d4aaebca42f8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" Nov 03 17:45:44.538757 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.538739 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/78a8038b-489c-44f5-827f-d4aaebca42f8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-c5q4r\" (UID: \"78a8038b-489c-44f5-827f-d4aaebca42f8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" Nov 03 17:45:44.551500 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.551479 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sf77n"] Nov 03 17:45:44.553683 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.553667 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.565038 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.565018 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 17:45:44.565123 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.565039 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 17:45:44.565123 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.565112 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-vlppl\"" Nov 03 17:45:44.565343 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.565310 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 17:45:44.569801 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.569775 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc"] Nov 03 17:45:44.611066 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.611035 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sm4t\" (UniqueName: \"kubernetes.io/projected/78a8038b-489c-44f5-827f-d4aaebca42f8-kube-api-access-9sm4t\") pod \"openshift-state-metrics-7bd7489957-c5q4r\" (UID: \"78a8038b-489c-44f5-827f-d4aaebca42f8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" Nov 03 17:45:44.636484 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.636450 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/83572d76-31ae-42e0-b7d4-acc08df84a9c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:45:44.636612 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.636492 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83572d76-31ae-42e0-b7d4-acc08df84a9c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:45:44.636612 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.636524 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb74w\" (UniqueName: \"kubernetes.io/projected/a8abced1-c904-4e84-b62e-96290db6fa8e-kube-api-access-tb74w\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.636612 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.636566 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83572d76-31ae-42e0-b7d4-acc08df84a9c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:45:44.636612 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.636604 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-accelerators-collector-config\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.636820 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.636669 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.636820 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.636765 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnv9j\" (UniqueName: \"kubernetes.io/projected/83572d76-31ae-42e0-b7d4-acc08df84a9c-kube-api-access-mnv9j\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:45:44.636820 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.636795 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8abced1-c904-4e84-b62e-96290db6fa8e-sys\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.636942 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.636833 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/83572d76-31ae-42e0-b7d4-acc08df84a9c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:45:44.637028 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.637007 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-tls\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.637096 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.637047 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-wtmp\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.637096 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.637070 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8abced1-c904-4e84-b62e-96290db6fa8e-metrics-client-ca\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.637190 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.637107 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-textfile\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.637190 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.637161 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/83572d76-31ae-42e0-b7d4-acc08df84a9c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:45:44.637190 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.637179 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/83572d76-31ae-42e0-b7d4-acc08df84a9c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:45:44.637328 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.637262 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8abced1-c904-4e84-b62e-96290db6fa8e-root\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.637378 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.637357 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83572d76-31ae-42e0-b7d4-acc08df84a9c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:45:44.637499 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.637483 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/83572d76-31ae-42e0-b7d4-acc08df84a9c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:45:44.638915 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.638894 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83572d76-31ae-42e0-b7d4-acc08df84a9c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:45:44.639233 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.639219 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/83572d76-31ae-42e0-b7d4-acc08df84a9c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:45:44.652285 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.652228 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 17:45:44.657960 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.657940 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 17:45:44.658143 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.658128 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-556wq\"" Nov 03 17:45:44.658268 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.658249 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 17:45:44.698867 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.698841 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnv9j\" (UniqueName: \"kubernetes.io/projected/83572d76-31ae-42e0-b7d4-acc08df84a9c-kube-api-access-mnv9j\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:45:44.709657 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.709640 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" Nov 03 17:45:44.738073 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.738045 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-tls\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.738073 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.738076 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-wtmp\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.738288 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.738092 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8abced1-c904-4e84-b62e-96290db6fa8e-metrics-client-ca\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.738288 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.738107 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-textfile\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.738288 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.738131 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8abced1-c904-4e84-b62e-96290db6fa8e-root\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.738288 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.738163 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb74w\" (UniqueName: \"kubernetes.io/projected/a8abced1-c904-4e84-b62e-96290db6fa8e-kube-api-access-tb74w\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.738288 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.738215 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-accelerators-collector-config\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.738288 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.738261 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-wtmp\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.738593 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.738316 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.738593 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.738367 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8abced1-c904-4e84-b62e-96290db6fa8e-sys\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.738593 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.738457 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8abced1-c904-4e84-b62e-96290db6fa8e-sys\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.738593 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.738557 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8abced1-c904-4e84-b62e-96290db6fa8e-root\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.738782 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.738606 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-textfile\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.738896 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.738877 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-accelerators-collector-config\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.738952 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.738895 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8abced1-c904-4e84-b62e-96290db6fa8e-metrics-client-ca\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.740435 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.740411 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-tls\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.740541 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.740526 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.799286 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.799258 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb74w\" (UniqueName: \"kubernetes.io/projected/a8abced1-c904-4e84-b62e-96290db6fa8e-kube-api-access-tb74w\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.818868 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.818837 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:45:44.866889 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:44.866850 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:45:44.877245 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:45:44.877216 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8abced1_c904_4e84_b62e_96290db6fa8e.slice/crio-3031219dfae0399fd53b98cf4d137df1b581cdca1752aa95c95b31c8fd6f715a WatchSource:0}: Error finding container 3031219dfae0399fd53b98cf4d137df1b581cdca1752aa95c95b31c8fd6f715a: Status 404 returned error can't find the container with id 3031219dfae0399fd53b98cf4d137df1b581cdca1752aa95c95b31c8fd6f715a Nov 03 17:45:45.129966 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:45.129930 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r"] Nov 03 17:45:45.133211 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:45:45.133183 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a8038b_489c_44f5_827f_d4aaebca42f8.slice/crio-d84cbd4a9e96f2d71b3b5c106d214bb0de8bea018ee8faa25233073f3fb7df92 WatchSource:0}: Error finding container d84cbd4a9e96f2d71b3b5c106d214bb0de8bea018ee8faa25233073f3fb7df92: Status 404 returned error can't find the container with id d84cbd4a9e96f2d71b3b5c106d214bb0de8bea018ee8faa25233073f3fb7df92 Nov 03 17:45:45.189844 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:45.189822 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc"] Nov 03 17:45:45.194794 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:45:45.194766 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83572d76_31ae_42e0_b7d4_acc08df84a9c.slice/crio-7eef05cfbc71d8bb1a328dfc16ed154eef1b84b82e1c73a05bc7b8175170dd99 WatchSource:0}: Error finding container 7eef05cfbc71d8bb1a328dfc16ed154eef1b84b82e1c73a05bc7b8175170dd99: Status 404 returned error can't find the container with id 7eef05cfbc71d8bb1a328dfc16ed154eef1b84b82e1c73a05bc7b8175170dd99 Nov 03 17:45:45.311935 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:45.311899 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:45.333690 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:45.333670 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:45:45.402607 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:45.402525 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sf77n" event={"ID":"a8abced1-c904-4e84-b62e-96290db6fa8e","Type":"ContainerStarted","Data":"3031219dfae0399fd53b98cf4d137df1b581cdca1752aa95c95b31c8fd6f715a"} Nov 03 17:45:45.403686 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:45.403653 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" event={"ID":"83572d76-31ae-42e0-b7d4-acc08df84a9c","Type":"ContainerStarted","Data":"7eef05cfbc71d8bb1a328dfc16ed154eef1b84b82e1c73a05bc7b8175170dd99"} Nov 03 17:45:45.405264 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:45.405244 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" event={"ID":"78a8038b-489c-44f5-827f-d4aaebca42f8","Type":"ContainerStarted","Data":"ccb1c5a00e966135dad743a68bce8ae5c6ec97575c5f0fc91db0e3a3ac8104ca"} Nov 03 17:45:45.405355 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:45.405280 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" event={"ID":"78a8038b-489c-44f5-827f-d4aaebca42f8","Type":"ContainerStarted","Data":"6c6d2edb02cfe20449bac25a885526893adce94fbe91e155a39f4927f955fe60"} Nov 03 17:45:45.405355 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:45.405300 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" event={"ID":"78a8038b-489c-44f5-827f-d4aaebca42f8","Type":"ContainerStarted","Data":"d84cbd4a9e96f2d71b3b5c106d214bb0de8bea018ee8faa25233073f3fb7df92"} Nov 03 17:45:45.514678 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:45.514643 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:46.028620 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.028587 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:46.243440 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.243385 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:46.411817 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.411783 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sf77n" event={"ID":"a8abced1-c904-4e84-b62e-96290db6fa8e","Type":"ContainerStarted","Data":"603181325bf1708b995dc9c57b8b797ec0ff81577177bc5c5cfa29133f7bae26"} Nov 03 17:45:46.809856 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.809817 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 17:45:46.813245 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.813219 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.824699 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.824669 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:46.855769 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.855740 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4e2b3783-58ff-4bb7-b713-6da60023f5ef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.855884 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.855780 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e2b3783-58ff-4bb7-b713-6da60023f5ef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.855884 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.855815 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e2b3783-58ff-4bb7-b713-6da60023f5ef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.856002 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.855910 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.856063 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.855998 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-web-config\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.856063 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.856032 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.856063 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.856062 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-config-volume\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.856215 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.856099 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.856215 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.856126 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e2b3783-58ff-4bb7-b713-6da60023f5ef-config-out\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.856215 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.856148 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2b3783-58ff-4bb7-b713-6da60023f5ef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.856215 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.856169 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.856215 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.856188 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.856215 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.856202 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw4rq\" (UniqueName: \"kubernetes.io/projected/4e2b3783-58ff-4bb7-b713-6da60023f5ef-kube-api-access-hw4rq\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.867084 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.867062 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jtjm7\"" Nov 03 17:45:46.867191 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.867085 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 17:45:46.867191 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.867104 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 17:45:46.867191 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.867128 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 17:45:46.867191 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.867143 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 17:45:46.867191 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.867159 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 17:45:46.867191 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.867165 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 17:45:46.867191 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.867060 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 17:45:46.957173 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.957148 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.957321 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.957197 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-config-volume\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.957321 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.957235 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.957321 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.957260 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e2b3783-58ff-4bb7-b713-6da60023f5ef-config-out\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.957321 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.957276 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2b3783-58ff-4bb7-b713-6da60023f5ef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.957321 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.957302 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.957581 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.957331 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.957581 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.957354 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw4rq\" (UniqueName: \"kubernetes.io/projected/4e2b3783-58ff-4bb7-b713-6da60023f5ef-kube-api-access-hw4rq\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.957581 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.957377 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4e2b3783-58ff-4bb7-b713-6da60023f5ef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.957581 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.957406 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e2b3783-58ff-4bb7-b713-6da60023f5ef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.957581 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.957435 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e2b3783-58ff-4bb7-b713-6da60023f5ef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.957581 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.957462 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.957581 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.957516 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-web-config\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.957902 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.957827 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4e2b3783-58ff-4bb7-b713-6da60023f5ef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.960087 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.959926 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e2b3783-58ff-4bb7-b713-6da60023f5ef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.960529 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.960476 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.960529 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.960474 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-config-volume\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.960709 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.960686 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.961463 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.961443 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.961576 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.961466 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e2b3783-58ff-4bb7-b713-6da60023f5ef-config-out\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.961576 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.961509 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.962768 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.962746 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.962914 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.962895 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-web-config\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.968304 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.968269 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 17:45:46.968428 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.968364 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 17:45:46.969792 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.969770 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 17:45:46.971374 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.971356 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e2b3783-58ff-4bb7-b713-6da60023f5ef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:46.979019 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:46.978998 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2b3783-58ff-4bb7-b713-6da60023f5ef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:47.020672 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:47.020640 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw4rq\" (UniqueName: \"kubernetes.io/projected/4e2b3783-58ff-4bb7-b713-6da60023f5ef-kube-api-access-hw4rq\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:47.133242 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:47.133214 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:45:47.147598 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:47.147570 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:47.417087 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:47.416999 2574 generic.go:356] "Generic (PLEG): container finished" podID="a8abced1-c904-4e84-b62e-96290db6fa8e" containerID="603181325bf1708b995dc9c57b8b797ec0ff81577177bc5c5cfa29133f7bae26" exitCode=0 Nov 03 17:45:47.417519 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:47.417077 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sf77n" event={"ID":"a8abced1-c904-4e84-b62e-96290db6fa8e","Type":"ContainerDied","Data":"603181325bf1708b995dc9c57b8b797ec0ff81577177bc5c5cfa29133f7bae26"} Nov 03 17:45:47.419458 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:47.419438 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" event={"ID":"83572d76-31ae-42e0-b7d4-acc08df84a9c","Type":"ContainerStarted","Data":"13093aa5b353789990ab423fc8d302ac4a848d7ac5812dbe3f62c39c0e3acedf"} Nov 03 17:45:47.419545 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:47.419487 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" event={"ID":"83572d76-31ae-42e0-b7d4-acc08df84a9c","Type":"ContainerStarted","Data":"50829f18a8c7e844b1f422f1baf4f4bdd6d7d4f35145c9bc9ea47ff05da57e2f"} Nov 03 17:45:47.419545 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:47.419510 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" event={"ID":"83572d76-31ae-42e0-b7d4-acc08df84a9c","Type":"ContainerStarted","Data":"2576c114a2717bcd5d07426fbebe7c70d34c62c5438211bf78bbbe984393b2ad"} Nov 03 17:45:47.421459 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:47.421434 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" event={"ID":"78a8038b-489c-44f5-827f-d4aaebca42f8","Type":"ContainerStarted","Data":"2272e372683d8bd87de5f6cbaa0a222acb9007c71945b4990cfb53d9d881afed"} Nov 03 17:45:47.440407 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:47.440375 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:47.568365 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:47.567355 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 17:45:47.578836 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:45:47.578808 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e2b3783_58ff_4bb7_b713_6da60023f5ef.slice/crio-8417337ed283c083b428775224f475c51445e5bbc02329e40d863a669796181a WatchSource:0}: Error finding container 8417337ed283c083b428775224f475c51445e5bbc02329e40d863a669796181a: Status 404 returned error can't find the container with id 8417337ed283c083b428775224f475c51445e5bbc02329e40d863a669796181a Nov 03 17:45:47.668499 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:47.668369 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" podStartSLOduration=2.05280469 podStartE2EDuration="3.668356698s" totalImagesPullingTime="1.615552008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:47.667251899 +0000 UTC m=+58.011618276" watchObservedRunningTime="2026-11-03 17:45:47.668356698 +0000 UTC m=+58.012723074" Nov 03 17:45:47.675840 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:47.675805 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:47.789923 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:47.789854 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" podStartSLOduration=2.060884855 podStartE2EDuration="3.789833057s" totalImagesPullingTime="1.728948202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:47.788949468 +0000 UTC m=+58.133315848" watchObservedRunningTime="2026-11-03 17:45:47.789833057 +0000 UTC m=+58.134199435" Nov 03 17:45:47.874267 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:47.874204 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:48.426873 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:48.426841 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sf77n" event={"ID":"a8abced1-c904-4e84-b62e-96290db6fa8e","Type":"ContainerStarted","Data":"6f619770e2f20953ae5a9e5b7b90eea4a54dcadaebbfaec12c982ad34220b3b1"} Nov 03 17:45:48.427268 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:48.426890 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sf77n" event={"ID":"a8abced1-c904-4e84-b62e-96290db6fa8e","Type":"ContainerStarted","Data":"0a14b4a3b4a8b39ac4ad75c28f54d3ba889cc3215fd7fdab3aeecdf601cad233"} Nov 03 17:45:48.428116 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:48.428091 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e2b3783-58ff-4bb7-b713-6da60023f5ef","Type":"ContainerStarted","Data":"8417337ed283c083b428775224f475c51445e5bbc02329e40d863a669796181a"} Nov 03 17:45:48.547779 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:48.547723 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sf77n" podStartSLOduration=3.086077672 podStartE2EDuration="4.547709005s" totalImagesPullingTime="1.378883871s" totalInitContainerRuntime="82.747462ms" isStatefulPod=true podCreationTimestamp="2026-11-03 17:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:48.546727958 +0000 UTC m=+58.891094336" watchObservedRunningTime="2026-11-03 17:45:48.547709005 +0000 UTC m=+58.892075381" Nov 03 17:45:48.816544 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:48.816503 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:49.092875 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.092792 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:49.394690 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.394462 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:49.432558 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.432522 2574 generic.go:356] "Generic (PLEG): container finished" podID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerID="27a60c8c7c4610b479e0c7e24c5cf4dd2054502a61ed64d7bdb7e8f3751d87a9" exitCode=0 Nov 03 17:45:49.432955 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.432597 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e2b3783-58ff-4bb7-b713-6da60023f5ef","Type":"ContainerDied","Data":"27a60c8c7c4610b479e0c7e24c5cf4dd2054502a61ed64d7bdb7e8f3751d87a9"} Nov 03 17:45:49.470847 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.470818 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-687b6669ff-vc8sm"] Nov 03 17:45:49.472942 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.472923 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:45:49.525926 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.525894 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 17:45:49.526058 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.525938 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 17:45:49.526058 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.525984 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 17:45:49.526148 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.526115 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 17:45:49.526148 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.526130 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bungpse31jqe3\"" Nov 03 17:45:49.526205 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.526178 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-z8nlf\"" Nov 03 17:45:49.532581 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.532557 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-687b6669ff-vc8sm"] Nov 03 17:45:49.579233 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.579205 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d82f08c7-9da6-4969-858f-c8e5b140d27c-audit-log\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:45:49.579453 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.579248 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d82f08c7-9da6-4969-858f-c8e5b140d27c-secret-metrics-server-tls\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:45:49.579453 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.579270 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d82f08c7-9da6-4969-858f-c8e5b140d27c-metrics-server-audit-profiles\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:45:49.579453 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.579429 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d82f08c7-9da6-4969-858f-c8e5b140d27c-secret-metrics-server-client-certs\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:45:49.579453 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.579453 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d82f08c7-9da6-4969-858f-c8e5b140d27c-client-ca-bundle\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:45:49.579625 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.579470 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5m9x\" (UniqueName: \"kubernetes.io/projected/d82f08c7-9da6-4969-858f-c8e5b140d27c-kube-api-access-v5m9x\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:45:49.579625 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.579495 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d82f08c7-9da6-4969-858f-c8e5b140d27c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:45:49.604104 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.604064 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:49.680251 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.680168 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d82f08c7-9da6-4969-858f-c8e5b140d27c-secret-metrics-server-client-certs\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:45:49.680251 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.680205 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d82f08c7-9da6-4969-858f-c8e5b140d27c-client-ca-bundle\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:45:49.680251 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.680232 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5m9x\" (UniqueName: \"kubernetes.io/projected/d82f08c7-9da6-4969-858f-c8e5b140d27c-kube-api-access-v5m9x\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:45:49.680532 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.680269 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d82f08c7-9da6-4969-858f-c8e5b140d27c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:45:49.680532 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.680304 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d82f08c7-9da6-4969-858f-c8e5b140d27c-audit-log\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:45:49.680532 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.680433 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d82f08c7-9da6-4969-858f-c8e5b140d27c-secret-metrics-server-tls\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:45:49.680532 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.680472 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d82f08c7-9da6-4969-858f-c8e5b140d27c-metrics-server-audit-profiles\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:45:49.680834 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.680800 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d82f08c7-9da6-4969-858f-c8e5b140d27c-audit-log\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:45:49.681096 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.681071 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d82f08c7-9da6-4969-858f-c8e5b140d27c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:45:49.681436 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.681415 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d82f08c7-9da6-4969-858f-c8e5b140d27c-metrics-server-audit-profiles\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:45:49.682945 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.682922 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d82f08c7-9da6-4969-858f-c8e5b140d27c-secret-metrics-server-tls\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:45:49.683247 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.683232 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d82f08c7-9da6-4969-858f-c8e5b140d27c-client-ca-bundle\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:45:49.683360 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.683340 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d82f08c7-9da6-4969-858f-c8e5b140d27c-secret-metrics-server-client-certs\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:45:49.741589 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.741564 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5m9x\" (UniqueName: \"kubernetes.io/projected/d82f08c7-9da6-4969-858f-c8e5b140d27c-kube-api-access-v5m9x\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:45:49.783676 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:49.783651 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:45:50.164141 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.164111 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-67d8bb587d-lcgff"] Nov 03 17:45:50.167414 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.167371 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.209469 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.209439 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-687b6669ff-vc8sm"] Nov 03 17:45:50.219946 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:45:50.219920 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd82f08c7_9da6_4969_858f_c8e5b140d27c.slice/crio-e837bd97a0ae1a25f103e74c16ee1958b0b28214340fc5062648f760b906d683 WatchSource:0}: Error finding container e837bd97a0ae1a25f103e74c16ee1958b0b28214340fc5062648f760b906d683: Status 404 returned error can't find the container with id e837bd97a0ae1a25f103e74c16ee1958b0b28214340fc5062648f760b906d683 Nov 03 17:45:50.220683 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.220664 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 17:45:50.220768 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.220680 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 17:45:50.220768 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.220709 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 17:45:50.220862 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.220774 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-q2hxk\"" Nov 03 17:45:50.220862 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.220819 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 17:45:50.220948 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.220940 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 17:45:50.271216 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.271172 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 17:45:50.273208 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.273166 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-67d8bb587d-lcgff"] Nov 03 17:45:50.285924 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.285899 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c5890bb8-3e6a-4250-8ccb-50771cd112c0-federate-client-tls\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.286089 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.285951 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2c2r\" (UniqueName: \"kubernetes.io/projected/c5890bb8-3e6a-4250-8ccb-50771cd112c0-kube-api-access-l2c2r\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.286089 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.285987 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5890bb8-3e6a-4250-8ccb-50771cd112c0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.286089 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.286077 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c5890bb8-3e6a-4250-8ccb-50771cd112c0-telemeter-client-tls\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.286247 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.286146 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c5890bb8-3e6a-4250-8ccb-50771cd112c0-secret-telemeter-client\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.286247 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.286179 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5890bb8-3e6a-4250-8ccb-50771cd112c0-metrics-client-ca\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.286247 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.286200 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5890bb8-3e6a-4250-8ccb-50771cd112c0-serving-certs-ca-bundle\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.286247 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.286234 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5890bb8-3e6a-4250-8ccb-50771cd112c0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.387100 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.387073 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c5890bb8-3e6a-4250-8ccb-50771cd112c0-secret-telemeter-client\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.387272 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.387113 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5890bb8-3e6a-4250-8ccb-50771cd112c0-metrics-client-ca\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.387272 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.387133 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5890bb8-3e6a-4250-8ccb-50771cd112c0-serving-certs-ca-bundle\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.387272 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.387154 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5890bb8-3e6a-4250-8ccb-50771cd112c0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.387272 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.387176 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c5890bb8-3e6a-4250-8ccb-50771cd112c0-federate-client-tls\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.387272 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.387207 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2c2r\" (UniqueName: \"kubernetes.io/projected/c5890bb8-3e6a-4250-8ccb-50771cd112c0-kube-api-access-l2c2r\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.387272 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.387232 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5890bb8-3e6a-4250-8ccb-50771cd112c0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.387272 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.387268 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c5890bb8-3e6a-4250-8ccb-50771cd112c0-telemeter-client-tls\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.387989 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.387960 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5890bb8-3e6a-4250-8ccb-50771cd112c0-metrics-client-ca\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.388086 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.388008 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5890bb8-3e6a-4250-8ccb-50771cd112c0-serving-certs-ca-bundle\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.388703 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.388681 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5890bb8-3e6a-4250-8ccb-50771cd112c0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.389887 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.389861 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5890bb8-3e6a-4250-8ccb-50771cd112c0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.390007 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.389987 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c5890bb8-3e6a-4250-8ccb-50771cd112c0-secret-telemeter-client\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.390490 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.390467 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c5890bb8-3e6a-4250-8ccb-50771cd112c0-telemeter-client-tls\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.390573 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.390507 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c5890bb8-3e6a-4250-8ccb-50771cd112c0-federate-client-tls\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.396235 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.396212 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-khc2s" Nov 03 17:45:50.436448 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.436358 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" event={"ID":"d82f08c7-9da6-4969-858f-c8e5b140d27c","Type":"ContainerStarted","Data":"e837bd97a0ae1a25f103e74c16ee1958b0b28214340fc5062648f760b906d683"} Nov 03 17:45:50.449166 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.449138 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2c2r\" (UniqueName: \"kubernetes.io/projected/c5890bb8-3e6a-4250-8ccb-50771cd112c0-kube-api-access-l2c2r\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.479857 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.479812 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:45:50.919769 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:50.919733 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-67d8bb587d-lcgff"] Nov 03 17:45:50.923697 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:45:50.923663 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5890bb8_3e6a_4250_8ccb_50771cd112c0.slice/crio-e90efb6b9b1043c03eb07217f59b0009c5403b452caf89e81ff6926af9673b50 WatchSource:0}: Error finding container e90efb6b9b1043c03eb07217f59b0009c5403b452caf89e81ff6926af9673b50: Status 404 returned error can't find the container with id e90efb6b9b1043c03eb07217f59b0009c5403b452caf89e81ff6926af9673b50 Nov 03 17:45:51.252022 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:51.251934 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:51.441994 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:51.441956 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" event={"ID":"c5890bb8-3e6a-4250-8ccb-50771cd112c0","Type":"ContainerStarted","Data":"e90efb6b9b1043c03eb07217f59b0009c5403b452caf89e81ff6926af9673b50"} Nov 03 17:45:51.458978 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:51.458920 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:52.447737 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:52.447691 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" event={"ID":"d82f08c7-9da6-4969-858f-c8e5b140d27c","Type":"ContainerStarted","Data":"6016d0e9b10f84204db3d374f683503cdf4958fb49668a44ee4dcbe151f305b7"} Nov 03 17:45:52.567941 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:52.567883 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" podStartSLOduration=1.835451152 podStartE2EDuration="3.567859516s" totalImagesPullingTime="1.732408364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:52.565910943 +0000 UTC m=+62.910277538" watchObservedRunningTime="2026-11-03 17:45:52.567859516 +0000 UTC m=+62.912225895" Nov 03 17:45:53.453568 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.453518 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" event={"ID":"c5890bb8-3e6a-4250-8ccb-50771cd112c0","Type":"ContainerStarted","Data":"b94edcc59117b2b3a2938232b87e1d8ffa5b83d68f98a359ce90c6fdec31f5cb"} Nov 03 17:45:53.454038 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.453608 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" event={"ID":"c5890bb8-3e6a-4250-8ccb-50771cd112c0","Type":"ContainerStarted","Data":"aa1a1291827c78fce278f5d0634cd998c9ebdf4d7232bbecb816d2c10bc125ee"} Nov 03 17:45:53.454038 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.453637 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" event={"ID":"c5890bb8-3e6a-4250-8ccb-50771cd112c0","Type":"ContainerStarted","Data":"b4698201325001bea94848e02e21294c1f804a8223a445b85f3a265632bc7a62"} Nov 03 17:45:53.456769 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.456739 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e2b3783-58ff-4bb7-b713-6da60023f5ef","Type":"ContainerStarted","Data":"cedf34711d8e93cd535cb861ff9d77df849558d9a2e82aa9802f50f293de0536"} Nov 03 17:45:53.456901 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.456796 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e2b3783-58ff-4bb7-b713-6da60023f5ef","Type":"ContainerStarted","Data":"8a9f470a2e6a404567af040875d8c349376de276d7f0abab1f4d0860556493af"} Nov 03 17:45:53.456901 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.456824 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e2b3783-58ff-4bb7-b713-6da60023f5ef","Type":"ContainerStarted","Data":"ee72547fd301caeabc0595fa2f411ed58434fd7f5f6607186969b2b82a87d0ad"} Nov 03 17:45:53.456901 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.456847 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e2b3783-58ff-4bb7-b713-6da60023f5ef","Type":"ContainerStarted","Data":"28f766fde6c28ff9b996544936025fd9294d28b2bb76271ba7c3c61bee954dab"} Nov 03 17:45:53.456901 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.456872 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e2b3783-58ff-4bb7-b713-6da60023f5ef","Type":"ContainerStarted","Data":"b1b9657c8a13ebb7a0205c090d1b46ae65244bf207919d8590ac7bce749abdd3"} Nov 03 17:45:53.578428 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.578363 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" podStartSLOduration=1.450615784 podStartE2EDuration="3.578349214s" totalImagesPullingTime="2.12773343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:53.576939906 +0000 UTC m=+63.921306278" watchObservedRunningTime="2026-11-03 17:45:53.578349214 +0000 UTC m=+63.922715591" Nov 03 17:45:53.789963 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.789885 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 17:45:53.793017 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.792998 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:53.810932 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.810899 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:53.846764 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.846737 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 17:45:53.846906 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.846779 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 17:45:53.846906 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.846835 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 17:45:53.847216 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.847190 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 17:45:53.847317 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.847246 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 17:45:53.847317 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.847252 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 17:45:53.847317 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.847294 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 17:45:53.847495 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.847328 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 17:45:53.847495 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.847356 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-251ja618v50s1\"" Nov 03 17:45:53.847967 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.847949 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 17:45:53.848155 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.848138 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 17:45:53.897019 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.896991 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7d5fs\"" Nov 03 17:45:53.897158 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.897041 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 17:45:53.897752 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.897710 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 17:45:53.915867 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.915844 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:53.915992 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.915874 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:53.915992 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.915907 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:53.915992 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.915962 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-config\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:53.916097 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.916023 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-web-config\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:53.916097 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.916039 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnmr6\" (UniqueName: \"kubernetes.io/projected/604769b0-503d-444f-983e-265898035e90-kube-api-access-xnmr6\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:53.916097 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.916062 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:53.916097 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.916089 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:53.916213 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.916151 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/604769b0-503d-444f-983e-265898035e90-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:53.916213 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.916182 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:53.916213 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.916201 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:53.916293 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.916220 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:53.916293 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.916271 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:53.916347 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.916298 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:53.916347 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.916340 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:53.916434 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.916355 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:53.916434 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.916383 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:53.916497 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.916436 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:53.916525 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.916495 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:53.916554 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.916523 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/604769b0-503d-444f-983e-265898035e90-config-out\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:53.947843 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.947820 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 17:45:53.947962 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.947882 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 17:45:53.949576 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:53.949556 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 17:45:54.017324 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.017296 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/604769b0-503d-444f-983e-265898035e90-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.017475 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.017350 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.017475 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.017381 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.017475 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.017431 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.017475 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.017456 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.017675 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.017480 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.017675 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.017530 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.017675 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.017555 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.017675 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.017603 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.017675 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.017634 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.017675 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.017667 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.017944 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.017702 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/604769b0-503d-444f-983e-265898035e90-config-out\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.017944 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.017733 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.017944 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.017758 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.017944 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.017782 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.017944 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.017799 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-config\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.017944 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.017825 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-web-config\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.017944 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.017854 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnmr6\" (UniqueName: \"kubernetes.io/projected/604769b0-503d-444f-983e-265898035e90-kube-api-access-xnmr6\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.017944 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.017895 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.017944 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.017927 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.022808 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.019152 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.022808 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.019269 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.022808 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.020682 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.022808 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.020989 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.022808 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.021241 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.022808 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.021412 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.022808 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.021468 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.022808 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.021506 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/604769b0-503d-444f-983e-265898035e90-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.022808 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.021578 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.022808 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.021791 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.022808 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.022119 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.022808 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.022768 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.023375 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.022926 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.023672 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.023647 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.023801 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.023703 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/604769b0-503d-444f-983e-265898035e90-config-out\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.023864 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.023802 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.024132 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.024108 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-web-config\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.024132 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.024122 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-config\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.024731 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.024708 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.082915 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.082882 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnmr6\" (UniqueName: \"kubernetes.io/projected/604769b0-503d-444f-983e-265898035e90-kube-api-access-xnmr6\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.111795 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.111764 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:45:54.463167 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.463089 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e2b3783-58ff-4bb7-b713-6da60023f5ef","Type":"ContainerStarted","Data":"53cb5b2140417db45fdf8b53a2d7689731199c3df0ac1a7758f4357e3c344bfb"} Nov 03 17:45:54.549027 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.548913 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 17:45:54.551563 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:45:54.551537 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604769b0_503d_444f_983e_265898035e90.slice/crio-4d74568c685e0144ec9932191f629f76220da113cbdff3176f508cc6984c07c2 WatchSource:0}: Error finding container 4d74568c685e0144ec9932191f629f76220da113cbdff3176f508cc6984c07c2: Status 404 returned error can't find the container with id 4d74568c685e0144ec9932191f629f76220da113cbdff3176f508cc6984c07c2 Nov 03 17:45:54.593691 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:54.593649 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.704919776 podStartE2EDuration="8.593636625s" totalImagesPullingTime="5.877273139s" totalInitContainerRuntime="11.44371ms" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:54.592187454 +0000 UTC m=+64.936553828" watchObservedRunningTime="2026-11-03 17:45:54.593636625 +0000 UTC m=+64.938003002" Nov 03 17:45:55.468282 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:55.468245 2574 generic.go:356] "Generic (PLEG): container finished" podID="604769b0-503d-444f-983e-265898035e90" containerID="15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2" exitCode=0 Nov 03 17:45:55.468793 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:55.468343 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"604769b0-503d-444f-983e-265898035e90","Type":"ContainerDied","Data":"15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2"} Nov 03 17:45:55.468793 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:55.468460 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"604769b0-503d-444f-983e-265898035e90","Type":"ContainerStarted","Data":"4d74568c685e0144ec9932191f629f76220da113cbdff3176f508cc6984c07c2"} Nov 03 17:45:55.716758 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:55.716729 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:55.985290 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:55.985252 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:56.036978 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:56.036947 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret\") pod \"global-pull-secret-syncer-8zpkh\" (UID: \"c527a13f-da23-4228-a048-4af962c36340\") " pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:56.037132 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:56.037036 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs\") pod \"network-metrics-daemon-fs77w\" (UID: \"0ba1ef12-3f5d-4449-8125-4a1921ec2990\") " pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:56.089104 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:56.089081 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 17:45:56.089234 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:56.089113 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 17:45:56.099375 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:56.099356 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs\") pod \"network-metrics-daemon-fs77w\" (UID: \"0ba1ef12-3f5d-4449-8125-4a1921ec2990\") " pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:56.099481 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:56.099376 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret\") pod \"global-pull-secret-syncer-8zpkh\" (UID: \"c527a13f-da23-4228-a048-4af962c36340\") " pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:56.137727 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:56.137706 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48n9p\" (UniqueName: \"kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p\") pod \"network-check-target-8p8q8\" (UID: \"d2c0ef79-ea05-4061-b4dd-31edac551213\") " pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:56.189689 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:56.189669 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 17:45:56.250553 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:56.250495 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 17:45:56.261803 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:56.261771 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48n9p\" (UniqueName: \"kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p\") pod \"network-check-target-8p8q8\" (UID: \"d2c0ef79-ea05-4061-b4dd-31edac551213\") " pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:56.343660 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:56.343635 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6lzh9\"" Nov 03 17:45:56.347589 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:56.347574 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sq9vz\"" Nov 03 17:45:56.352669 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:56.352654 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:45:56.353772 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:56.353755 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f85t9\"" Nov 03 17:45:56.356523 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:56.356506 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:45:56.363166 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:56.363144 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:45:56.735658 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:56.735425 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8zpkh"] Nov 03 17:45:56.737910 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:45:56.737876 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc527a13f_da23_4228_a048_4af962c36340.slice/crio-bac3983211e839f7b74fd7fdbd1e2c9a0f1bcdc526035eb349559a320bda65f9 WatchSource:0}: Error finding container bac3983211e839f7b74fd7fdbd1e2c9a0f1bcdc526035eb349559a320bda65f9: Status 404 returned error can't find the container with id bac3983211e839f7b74fd7fdbd1e2c9a0f1bcdc526035eb349559a320bda65f9 Nov 03 17:45:56.787629 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:56.787606 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8p8q8"] Nov 03 17:45:56.790269 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:45:56.790248 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c0ef79_ea05_4061_b4dd_31edac551213.slice/crio-976df117fe7139e172bcc029b789632d77154ac99f193d221dd5b15cf390e9ea WatchSource:0}: Error finding container 976df117fe7139e172bcc029b789632d77154ac99f193d221dd5b15cf390e9ea: Status 404 returned error can't find the container with id 976df117fe7139e172bcc029b789632d77154ac99f193d221dd5b15cf390e9ea Nov 03 17:45:56.791648 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:56.791619 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fs77w"] Nov 03 17:45:56.794231 ip-10-0-137-164 kubenswrapper[2574]: W1103 17:45:56.794208 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba1ef12_3f5d_4449_8125_4a1921ec2990.slice/crio-1df7518f89cd7da9871f6d8d169e25bd3f80e19ae487165c824c1b3bec821187 WatchSource:0}: Error finding container 1df7518f89cd7da9871f6d8d169e25bd3f80e19ae487165c824c1b3bec821187: Status 404 returned error can't find the container with id 1df7518f89cd7da9871f6d8d169e25bd3f80e19ae487165c824c1b3bec821187 Nov 03 17:45:57.477903 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:57.477857 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fs77w" event={"ID":"0ba1ef12-3f5d-4449-8125-4a1921ec2990","Type":"ContainerStarted","Data":"1df7518f89cd7da9871f6d8d169e25bd3f80e19ae487165c824c1b3bec821187"} Nov 03 17:45:57.479235 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:57.479202 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8p8q8" event={"ID":"d2c0ef79-ea05-4061-b4dd-31edac551213","Type":"ContainerStarted","Data":"976df117fe7139e172bcc029b789632d77154ac99f193d221dd5b15cf390e9ea"} Nov 03 17:45:57.480469 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:57.480441 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8zpkh" event={"ID":"c527a13f-da23-4228-a048-4af962c36340","Type":"ContainerStarted","Data":"bac3983211e839f7b74fd7fdbd1e2c9a0f1bcdc526035eb349559a320bda65f9"} Nov 03 17:45:58.214477 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:58.214444 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:45:59.495090 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:59.495058 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fs77w" event={"ID":"0ba1ef12-3f5d-4449-8125-4a1921ec2990","Type":"ContainerStarted","Data":"08b2c64d17eeb4d882ae94f2c979d3f66c8d0c01ae76c6d37d62cbcae7a6895b"} Nov 03 17:45:59.495565 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:59.495140 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fs77w" event={"ID":"0ba1ef12-3f5d-4449-8125-4a1921ec2990","Type":"ContainerStarted","Data":"3b94427b106eb98f3afea65c213731e30b4ea762a9ffb126971027e315a09e9f"} Nov 03 17:45:59.614272 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:45:59.614201 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fs77w" podStartSLOduration=67.976935685 podStartE2EDuration="1m9.614181739s" totalImagesPullingTime="1.637246054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:59.612848488 +0000 UTC m=+69.957214865" watchObservedRunningTime="2026-11-03 17:45:59.614181739 +0000 UTC m=+69.958548118" Nov 03 17:46:00.045378 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:46:00.045344 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:00.285540 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:46:00.285344 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:00.502483 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:46:00.502299 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"604769b0-503d-444f-983e-265898035e90","Type":"ContainerStarted","Data":"78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562"} Nov 03 17:46:00.502483 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:46:00.502407 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"604769b0-503d-444f-983e-265898035e90","Type":"ContainerStarted","Data":"5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2"} Nov 03 17:46:00.847297 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:46:00.847228 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:01.049435 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:46:01.049385 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:01.509351 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:46:01.509311 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8p8q8" event={"ID":"d2c0ef79-ea05-4061-b4dd-31edac551213","Type":"ContainerStarted","Data":"bdbe6a0b6ef92335166c5673407560f6f50249b43c1edfbb5f06ed1921c1df5b"} Nov 03 17:46:01.509749 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:46:01.509428 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:46:01.626054 ip-10-0-137-164 kubenswrapper[2574]: I1103 17:46:01.625996 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8p8q8" podStartSLOduration=67.019385792 podStartE2EDuration="1m11.625976479s" totalImagesPullingTime="4.606590687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:01.625450996 +0000 UTC m=+71.969817370" watchObservedRunningTime="2026-11-03 17:46:01.625976479 +0000 UTC m=+71.970342857" Nov 03 17:46:02.805945 ip-10-0-137-164 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 17:46:02.819038 ip-10-0-137-164 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 17:46:02.819198 ip-10-0-137-164 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 17:46:02.819244 ip-10-0-137-164 systemd[1]: kubelet.service: Consumed 4.320s CPU time. Nov 03 17:46:02.829043 ip-10-0-137-164 systemd[1]: Starting Kubernetes Kubelet... Nov 03 17:46:02.894820 ip-10-0-137-164 kubenswrapper[5805]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 17:46:02.894820 ip-10-0-137-164 kubenswrapper[5805]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 17:46:02.894820 ip-10-0-137-164 kubenswrapper[5805]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 17:46:02.894820 ip-10-0-137-164 kubenswrapper[5805]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 17:46:02.896866 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896789 5805 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:46:02.896866 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896865 5805 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:46:02.896866 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896869 5805 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:46:02.896960 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896873 5805 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:46:02.896960 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896876 5805 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:46:02.896960 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896879 5805 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:46:02.896960 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896882 5805 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:46:02.896960 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896885 5805 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:46:02.896960 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896887 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:46:02.896960 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896890 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:46:02.896960 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896893 5805 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:46:02.896960 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896895 5805 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:46:02.896960 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896898 5805 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:46:02.896960 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896901 5805 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:46:02.896960 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896904 5805 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:46:02.896960 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896906 5805 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:46:02.896960 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896909 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:46:02.896960 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896911 5805 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:46:02.896960 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896914 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:46:02.896960 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896917 5805 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:46:02.896960 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896919 5805 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:46:02.896960 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896923 5805 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:46:02.896960 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896926 5805 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:46:02.897456 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896928 5805 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:46:02.897456 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896931 5805 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:46:02.897456 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896933 5805 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:46:02.897456 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896942 5805 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:46:02.897456 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896945 5805 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:46:02.897456 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896948 5805 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:46:02.897456 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896950 5805 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:46:02.897456 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896954 5805 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:46:02.897456 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896956 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:46:02.897456 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896959 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:46:02.897456 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896962 5805 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:46:02.897456 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896965 5805 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:46:02.897456 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896967 5805 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:46:02.897456 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896970 5805 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:46:02.897456 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896973 5805 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:46:02.897456 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896976 5805 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:46:02.897456 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896979 5805 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:46:02.897456 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896982 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:46:02.897456 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896984 5805 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:46:02.897456 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896987 5805 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:46:02.897936 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896990 5805 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:46:02.897936 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896992 5805 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:46:02.897936 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896995 5805 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:46:02.897936 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.896997 5805 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:46:02.897936 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897000 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:46:02.897936 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897003 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:46:02.897936 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897007 5805 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:46:02.897936 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897011 5805 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:46:02.897936 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897014 5805 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:46:02.897936 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897016 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:46:02.897936 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897020 5805 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:46:02.897936 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897023 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:46:02.897936 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897026 5805 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:46:02.897936 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897028 5805 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:46:02.897936 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897031 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:46:02.897936 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897034 5805 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:46:02.897936 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897038 5805 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:46:02.897936 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897040 5805 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:46:02.897936 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897043 5805 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:46:02.898411 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897045 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:46:02.898411 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897048 5805 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:46:02.898411 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897051 5805 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:46:02.898411 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897054 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:46:02.898411 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897057 5805 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:46:02.898411 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897060 5805 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:46:02.898411 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897062 5805 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:46:02.898411 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897065 5805 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:46:02.898411 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897068 5805 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:46:02.898411 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897070 5805 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:46:02.898411 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897073 5805 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:46:02.898411 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897075 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:46:02.898411 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897078 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:46:02.898411 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897081 5805 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:46:02.898411 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897084 5805 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:46:02.898411 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897087 5805 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:46:02.898411 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897089 5805 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:46:02.898411 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897092 5805 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:46:02.898411 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897095 5805 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:46:02.898896 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897097 5805 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:46:02.898896 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897099 5805 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:46:02.898896 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897104 5805 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:46:02.898896 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897108 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:46:02.898896 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897111 5805 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:46:02.898896 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897114 5805 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:46:02.898896 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897116 5805 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:46:02.898896 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897119 5805 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:46:02.898896 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897121 5805 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:46:02.898896 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897124 5805 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:46:02.898896 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897127 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:46:02.898896 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897132 5805 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:46:02.898896 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897135 5805 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:46:02.898896 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897138 5805 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:46:02.898896 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897141 5805 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:46:02.898896 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897143 5805 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:46:02.898896 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897146 5805 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:46:02.898896 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897149 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:46:02.898896 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897152 5805 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:46:02.899330 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897155 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:46:02.899330 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897157 5805 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:46:02.899330 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897160 5805 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:46:02.899330 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897162 5805 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:46:02.899330 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897165 5805 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:46:02.899330 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897167 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:46:02.899330 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897986 5805 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:46:02.899330 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897994 5805 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:46:02.899330 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.897997 5805 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:46:02.899330 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898000 5805 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:46:02.899330 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898004 5805 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:46:02.899330 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898006 5805 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:46:02.899330 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898009 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:46:02.899330 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898012 5805 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:46:02.899330 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898014 5805 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:46:02.899330 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898017 5805 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:46:02.899330 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898019 5805 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:46:02.899330 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898022 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:46:02.899330 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898024 5805 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:46:02.899330 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898027 5805 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:46:02.899809 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898043 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:46:02.899809 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898047 5805 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:46:02.899809 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898052 5805 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:46:02.899809 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898055 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:46:02.899809 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898058 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:46:02.899809 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898066 5805 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:46:02.899809 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898069 5805 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:46:02.899809 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898072 5805 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:46:02.899809 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898075 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:46:02.899809 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898078 5805 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:46:02.899809 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898080 5805 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:46:02.899809 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898083 5805 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:46:02.899809 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898086 5805 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:46:02.899809 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898089 5805 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:46:02.899809 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898091 5805 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:46:02.899809 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898094 5805 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:46:02.899809 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898096 5805 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:46:02.899809 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898099 5805 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:46:02.899809 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898102 5805 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:46:02.900275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898104 5805 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:46:02.900275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898107 5805 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:46:02.900275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898109 5805 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:46:02.900275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898112 5805 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:46:02.900275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898114 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:46:02.900275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898117 5805 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:46:02.900275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898120 5805 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:46:02.900275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898122 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:46:02.900275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898124 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:46:02.900275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898127 5805 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:46:02.900275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898130 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:46:02.900275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898133 5805 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:46:02.900275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898136 5805 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:46:02.900275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898138 5805 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:46:02.900275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898141 5805 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:46:02.900275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898144 5805 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:46:02.900275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898146 5805 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:46:02.900275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898149 5805 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:46:02.900275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898152 5805 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:46:02.900275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898160 5805 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:46:02.900765 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898163 5805 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:46:02.900765 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898165 5805 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:46:02.900765 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898168 5805 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:46:02.900765 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898170 5805 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:46:02.900765 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898174 5805 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:46:02.900765 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898178 5805 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:46:02.900765 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898181 5805 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:46:02.900765 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898186 5805 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:46:02.900765 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898190 5805 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:46:02.900765 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898193 5805 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:46:02.900765 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898196 5805 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:46:02.900765 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898199 5805 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:46:02.900765 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898202 5805 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:46:02.900765 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898204 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:46:02.900765 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898207 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:46:02.900765 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898210 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:46:02.900765 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898213 5805 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:46:02.900765 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898216 5805 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:46:02.900765 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898218 5805 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:46:02.901212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898221 5805 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:46:02.901212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898224 5805 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:46:02.901212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898227 5805 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:46:02.901212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898230 5805 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:46:02.901212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898233 5805 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:46:02.901212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898235 5805 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:46:02.901212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898246 5805 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:46:02.901212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898249 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:46:02.901212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898251 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:46:02.901212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898254 5805 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:46:02.901212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898257 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:46:02.901212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898260 5805 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:46:02.901212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898263 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:46:02.901212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898273 5805 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:46:02.901212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898276 5805 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:46:02.901212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898279 5805 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:46:02.901212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898281 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:46:02.901212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898284 5805 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:46:02.901212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898287 5805 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:46:02.901720 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898290 5805 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:46:02.901720 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898292 5805 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:46:02.901720 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898295 5805 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:46:02.901720 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898298 5805 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:46:02.901720 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898301 5805 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:46:02.901720 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898306 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:46:02.901720 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898309 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:46:02.901720 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898312 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:46:02.901720 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898315 5805 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:46:02.901720 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898317 5805 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:46:02.901720 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898320 5805 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:46:02.901720 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898322 5805 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:46:02.901720 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898325 5805 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:46:02.901720 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898328 5805 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:46:02.901720 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898330 5805 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:46:02.901720 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898808 5805 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:46:02.901720 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898815 5805 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:46:02.901720 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898819 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:46:02.901720 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898821 5805 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:46:02.902172 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898824 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:46:02.902172 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898827 5805 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:46:02.902172 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898829 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:46:02.902172 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898832 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:46:02.902172 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898835 5805 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:46:02.902172 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898837 5805 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:46:02.902172 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898840 5805 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:46:02.902172 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898843 5805 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:46:02.902172 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898846 5805 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:46:02.902172 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898849 5805 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:46:02.902172 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898852 5805 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:46:02.902172 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898854 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:46:02.902172 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898857 5805 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:46:02.902172 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898860 5805 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:46:02.902172 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898862 5805 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:46:02.902172 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898865 5805 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:46:02.902172 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898868 5805 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:46:02.902172 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898871 5805 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:46:02.902172 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898874 5805 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:46:02.902172 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898877 5805 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:46:02.902710 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898879 5805 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:46:02.902710 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898882 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:46:02.902710 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898884 5805 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:46:02.902710 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898887 5805 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:46:02.902710 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898889 5805 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:46:02.902710 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898892 5805 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:46:02.902710 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898894 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:46:02.902710 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898897 5805 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:46:02.902710 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898900 5805 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:46:02.902710 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898902 5805 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:46:02.902710 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898905 5805 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:46:02.902710 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898908 5805 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:46:02.902710 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898910 5805 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:46:02.902710 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898913 5805 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:46:02.902710 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898915 5805 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:46:02.902710 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898918 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:46:02.902710 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898921 5805 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:46:02.902710 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898923 5805 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:46:02.902710 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898925 5805 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:46:02.902710 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898928 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:46:02.903440 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898931 5805 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:46:02.903440 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898934 5805 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:46:02.903440 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898936 5805 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:46:02.903440 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898939 5805 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:46:02.903440 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898942 5805 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:46:02.903440 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898944 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:46:02.903440 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898947 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:46:02.903440 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898950 5805 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:46:02.903440 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898952 5805 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:46:02.903440 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898957 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:46:02.903440 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898959 5805 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:46:02.903440 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898962 5805 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:46:02.903440 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898965 5805 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:46:02.903440 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898967 5805 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:46:02.903440 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898970 5805 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:46:02.903440 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898972 5805 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:46:02.903440 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898975 5805 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:46:02.903440 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898977 5805 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:46:02.903440 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898980 5805 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:46:02.903440 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898982 5805 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:46:02.904187 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898985 5805 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:46:02.904187 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898987 5805 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:46:02.904187 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898990 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:46:02.904187 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898992 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:46:02.904187 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898995 5805 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:46:02.904187 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.898999 5805 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:46:02.904187 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899002 5805 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:46:02.904187 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899005 5805 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:46:02.904187 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899008 5805 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:46:02.904187 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899011 5805 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:46:02.904187 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899013 5805 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:46:02.904187 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899016 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:46:02.904187 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899019 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:46:02.904187 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899022 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:46:02.904187 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899024 5805 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:46:02.904187 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899027 5805 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:46:02.904187 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899029 5805 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:46:02.904187 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899032 5805 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:46:02.904187 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899035 5805 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:46:02.904667 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899037 5805 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:46:02.904667 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899040 5805 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:46:02.904667 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899044 5805 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:46:02.904667 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899047 5805 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:46:02.904667 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899049 5805 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:46:02.904667 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899052 5805 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:46:02.904667 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899054 5805 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:46:02.904667 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899057 5805 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:46:02.904667 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899059 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:46:02.904667 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899062 5805 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:46:02.904667 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899064 5805 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:46:02.904667 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899067 5805 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:46:02.904667 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899069 5805 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:46:02.904667 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899072 5805 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:46:02.904667 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899074 5805 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:46:02.904667 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899077 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:46:02.904667 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899079 5805 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:46:02.904667 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899082 5805 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:46:02.904667 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899085 5805 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:46:02.904667 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899087 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:46:02.905140 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899090 5805 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:46:02.905140 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899094 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:46:02.905140 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899096 5805 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:46:02.905140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899195 5805 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 17:46:02.905140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899205 5805 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 17:46:02.905140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899215 5805 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 17:46:02.905140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899222 5805 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 17:46:02.905140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899229 5805 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 17:46:02.905140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899232 5805 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 17:46:02.905140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899236 5805 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 17:46:02.905140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899243 5805 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 17:46:02.905140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899246 5805 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 17:46:02.905140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899250 5805 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 17:46:02.905140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899253 5805 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 17:46:02.905140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899258 5805 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 17:46:02.905140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899261 5805 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 17:46:02.905140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899264 5805 flags.go:64] FLAG: --cgroup-root="" Nov 03 17:46:02.905140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899267 5805 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 17:46:02.905140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899270 5805 flags.go:64] FLAG: --client-ca-file="" Nov 03 17:46:02.905140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899273 5805 flags.go:64] FLAG: --cloud-provider="external" Nov 03 17:46:02.905140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899276 5805 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 17:46:02.905140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899280 5805 flags.go:64] FLAG: --cluster-domain="" Nov 03 17:46:02.905140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899282 5805 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 17:46:02.905140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899285 5805 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 17:46:02.905710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899289 5805 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 17:46:02.905710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899292 5805 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 17:46:02.905710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899296 5805 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 17:46:02.905710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899299 5805 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 17:46:02.905710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899302 5805 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 17:46:02.905710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899305 5805 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 17:46:02.905710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899308 5805 flags.go:64] FLAG: --contention-profiling="false" Nov 03 17:46:02.905710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899311 5805 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 17:46:02.905710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899314 5805 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 17:46:02.905710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899318 5805 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 17:46:02.905710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899321 5805 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 17:46:02.905710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899325 5805 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 17:46:02.905710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899328 5805 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 17:46:02.905710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899332 5805 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 17:46:02.905710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899335 5805 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 17:46:02.905710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899338 5805 flags.go:64] FLAG: --enable-server="true" Nov 03 17:46:02.905710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899342 5805 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 17:46:02.905710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899347 5805 flags.go:64] FLAG: --event-burst="100" Nov 03 17:46:02.905710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899351 5805 flags.go:64] FLAG: --event-qps="50" Nov 03 17:46:02.905710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899355 5805 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 17:46:02.905710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899358 5805 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 17:46:02.905710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899361 5805 flags.go:64] FLAG: --eviction-hard="" Nov 03 17:46:02.905710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899367 5805 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 17:46:02.905710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899370 5805 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 17:46:02.906287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899373 5805 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 17:46:02.906287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899376 5805 flags.go:64] FLAG: --eviction-soft="" Nov 03 17:46:02.906287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899379 5805 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 17:46:02.906287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899382 5805 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 17:46:02.906287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899385 5805 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 17:46:02.906287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899388 5805 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 17:46:02.906287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899402 5805 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 17:46:02.906287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899406 5805 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 17:46:02.906287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899409 5805 flags.go:64] FLAG: --feature-gates="" Nov 03 17:46:02.906287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899413 5805 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 17:46:02.906287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899416 5805 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 17:46:02.906287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899420 5805 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 17:46:02.906287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899423 5805 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 17:46:02.906287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899426 5805 flags.go:64] FLAG: --healthz-port="10248" Nov 03 17:46:02.906287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899429 5805 flags.go:64] FLAG: --help="false" Nov 03 17:46:02.906287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899432 5805 flags.go:64] FLAG: --hostname-override="ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:46:02.906287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899436 5805 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 17:46:02.906287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899439 5805 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 17:46:02.906287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899442 5805 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 17:46:02.906287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899445 5805 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 17:46:02.906287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899449 5805 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 17:46:02.906287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899452 5805 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 17:46:02.906287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899454 5805 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899457 5805 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899460 5805 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899464 5805 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899469 5805 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899472 5805 flags.go:64] FLAG: --kube-reserved="" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899475 5805 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899478 5805 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899483 5805 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899486 5805 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899489 5805 flags.go:64] FLAG: --lock-file="" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899491 5805 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899495 5805 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899498 5805 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899503 5805 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899506 5805 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899509 5805 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899511 5805 flags.go:64] FLAG: --logging-format="text" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899515 5805 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899518 5805 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899521 5805 flags.go:64] FLAG: --manifest-url="" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899524 5805 flags.go:64] FLAG: --manifest-url-header="" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899529 5805 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899532 5805 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899536 5805 flags.go:64] FLAG: --max-pods="110" Nov 03 17:46:02.906975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899540 5805 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899543 5805 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899546 5805 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899549 5805 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899552 5805 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899555 5805 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899558 5805 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899566 5805 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899569 5805 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899572 5805 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899575 5805 flags.go:64] FLAG: --pod-cidr="" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899579 5805 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899583 5805 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899586 5805 flags.go:64] FLAG: --pods-per-core="0" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899589 5805 flags.go:64] FLAG: --port="10250" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899594 5805 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899597 5805 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-07ee146735c7f22e8" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899600 5805 flags.go:64] FLAG: --qos-reserved="" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899603 5805 flags.go:64] FLAG: --read-only-port="10255" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899606 5805 flags.go:64] FLAG: --register-node="true" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899609 5805 flags.go:64] FLAG: --register-with-taints="" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899616 5805 flags.go:64] FLAG: --registry-burst="10" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899619 5805 flags.go:64] FLAG: --registry-qps="5" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899622 5805 flags.go:64] FLAG: --reserved-cpus="" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899625 5805 flags.go:64] FLAG: --reserved-memory="" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899629 5805 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 17:46:02.907567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899632 5805 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899636 5805 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899639 5805 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899642 5805 flags.go:64] FLAG: --runonce="false" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899645 5805 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899648 5805 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899651 5805 flags.go:64] FLAG: --seccomp-default="false" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899654 5805 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899657 5805 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899660 5805 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899663 5805 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899666 5805 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899669 5805 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899672 5805 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899675 5805 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899678 5805 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899681 5805 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899685 5805 flags.go:64] FLAG: --system-cgroups="" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899689 5805 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899694 5805 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899697 5805 flags.go:64] FLAG: --tls-cert-file="" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899702 5805 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899706 5805 flags.go:64] FLAG: --tls-min-version="" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899709 5805 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899712 5805 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 17:46:02.908222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899715 5805 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 17:46:02.908823 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899718 5805 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 17:46:02.908823 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899721 5805 flags.go:64] FLAG: --v="2" Nov 03 17:46:02.908823 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899725 5805 flags.go:64] FLAG: --version="false" Nov 03 17:46:02.908823 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899730 5805 flags.go:64] FLAG: --vmodule="" Nov 03 17:46:02.908823 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899733 5805 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 17:46:02.908823 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.899737 5805 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 17:46:02.908823 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899881 5805 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:46:02.908823 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899885 5805 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:46:02.908823 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899888 5805 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:46:02.908823 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899891 5805 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:46:02.908823 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899893 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:46:02.908823 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899896 5805 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:46:02.908823 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899899 5805 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:46:02.908823 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899901 5805 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:46:02.908823 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899904 5805 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:46:02.908823 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899906 5805 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:46:02.908823 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899909 5805 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:46:02.908823 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899911 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:46:02.908823 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899914 5805 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:46:02.908823 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899917 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:46:02.908823 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899919 5805 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:46:02.909428 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899922 5805 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:46:02.909428 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899925 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:46:02.909428 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899929 5805 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:46:02.909428 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899932 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:46:02.909428 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899941 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:46:02.909428 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899944 5805 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:46:02.909428 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899948 5805 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:46:02.909428 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899951 5805 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:46:02.909428 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899954 5805 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:46:02.909428 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899956 5805 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:46:02.909428 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899959 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:46:02.909428 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899962 5805 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:46:02.909428 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899965 5805 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:46:02.909428 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899967 5805 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:46:02.909428 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899970 5805 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:46:02.909428 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899973 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:46:02.909428 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899975 5805 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:46:02.909428 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899978 5805 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:46:02.909428 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899981 5805 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:46:02.910011 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899983 5805 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:46:02.910011 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899986 5805 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:46:02.910011 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899988 5805 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:46:02.910011 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899991 5805 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:46:02.910011 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899993 5805 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:46:02.910011 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899996 5805 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:46:02.910011 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.899999 5805 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:46:02.910011 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900001 5805 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:46:02.910011 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900004 5805 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:46:02.910011 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900007 5805 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:46:02.910011 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900009 5805 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:46:02.910011 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900012 5805 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:46:02.910011 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900015 5805 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:46:02.910011 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900017 5805 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:46:02.910011 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900020 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:46:02.910011 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900023 5805 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:46:02.910011 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900026 5805 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:46:02.910011 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900029 5805 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:46:02.910011 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900032 5805 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:46:02.910011 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900035 5805 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:46:02.910580 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900037 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:46:02.910580 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900040 5805 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:46:02.910580 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900042 5805 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:46:02.910580 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900045 5805 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:46:02.910580 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900047 5805 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:46:02.910580 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900050 5805 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:46:02.910580 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900053 5805 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:46:02.910580 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900055 5805 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:46:02.910580 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900058 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:46:02.910580 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900060 5805 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:46:02.910580 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900063 5805 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:46:02.910580 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900066 5805 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:46:02.910580 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900070 5805 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:46:02.910580 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900073 5805 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:46:02.910580 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900076 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:46:02.910580 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900080 5805 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:46:02.910580 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900083 5805 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:46:02.910580 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900085 5805 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:46:02.910580 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900088 5805 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:46:02.911097 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900090 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:46:02.911097 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900093 5805 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:46:02.911097 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900096 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:46:02.911097 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900098 5805 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:46:02.911097 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900101 5805 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:46:02.911097 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900105 5805 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:46:02.911097 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900108 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:46:02.911097 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900111 5805 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:46:02.911097 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900115 5805 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:46:02.911097 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900118 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:46:02.911097 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900121 5805 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:46:02.911097 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900124 5805 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:46:02.911097 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900127 5805 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:46:02.911097 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900130 5805 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:46:02.911097 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900132 5805 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:46:02.911097 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900135 5805 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:46:02.911097 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900137 5805 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:46:02.911097 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900140 5805 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:46:02.911097 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900142 5805 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:46:02.911568 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900145 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:46:02.911568 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900148 5805 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:46:02.911568 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900150 5805 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:46:02.911568 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900153 5805 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:46:02.911568 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900155 5805 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:46:02.911568 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900158 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:46:02.911568 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900161 5805 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:46:02.911568 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900163 5805 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:46:02.911568 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900166 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:46:02.911568 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900168 5805 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:46:02.911568 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900171 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:46:02.911568 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900174 5805 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:46:02.911568 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900176 5805 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:46:02.911568 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900179 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:46:02.911568 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.900205 5805 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 17:46:02.911568 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900243 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:46:02.911946 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900248 5805 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:46:02.911946 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900250 5805 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:46:02.911946 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900253 5805 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:46:02.911946 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900256 5805 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:46:02.911946 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900258 5805 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:46:02.911946 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900261 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:46:02.911946 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900265 5805 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:46:02.911946 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900268 5805 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:46:02.911946 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900271 5805 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:46:02.911946 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900274 5805 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:46:02.911946 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900279 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:46:02.911946 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900282 5805 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:46:02.911946 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900285 5805 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:46:02.911946 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900288 5805 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:46:02.911946 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900290 5805 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:46:02.911946 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900293 5805 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:46:02.911946 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900296 5805 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:46:02.911946 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900298 5805 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:46:02.911946 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900301 5805 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:46:02.911946 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900304 5805 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:46:02.912612 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900306 5805 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:46:02.912612 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900309 5805 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:46:02.912612 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900311 5805 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:46:02.912612 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900314 5805 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:46:02.912612 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900316 5805 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:46:02.912612 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900319 5805 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:46:02.912612 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900321 5805 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:46:02.912612 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900324 5805 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:46:02.912612 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900327 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:46:02.912612 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900330 5805 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:46:02.912612 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900332 5805 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:46:02.912612 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900335 5805 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:46:02.912612 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900337 5805 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:46:02.912612 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900340 5805 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:46:02.912612 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900342 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:46:02.912612 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900345 5805 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:46:02.912612 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900347 5805 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:46:02.912612 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900350 5805 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:46:02.912612 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900354 5805 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:46:02.913281 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900357 5805 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:46:02.913281 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900360 5805 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:46:02.913281 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900362 5805 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:46:02.913281 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900366 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:46:02.913281 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900369 5805 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:46:02.913281 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900372 5805 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:46:02.913281 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900374 5805 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:46:02.913281 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900378 5805 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:46:02.913281 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900382 5805 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:46:02.913281 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900386 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:46:02.913281 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900389 5805 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:46:02.913281 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900405 5805 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:46:02.913281 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900409 5805 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:46:02.913281 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900412 5805 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:46:02.913281 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900416 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:46:02.913281 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900418 5805 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:46:02.913281 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900422 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:46:02.913281 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900425 5805 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:46:02.913281 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900428 5805 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:46:02.913816 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900432 5805 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:46:02.913816 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900435 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:46:02.913816 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900438 5805 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:46:02.913816 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900441 5805 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:46:02.913816 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900444 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:46:02.913816 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900446 5805 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:46:02.913816 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900449 5805 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:46:02.913816 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900452 5805 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:46:02.913816 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900455 5805 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:46:02.913816 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900458 5805 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:46:02.913816 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900460 5805 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:46:02.913816 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900463 5805 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:46:02.913816 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900468 5805 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:46:02.913816 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900470 5805 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:46:02.913816 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900473 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:46:02.913816 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900476 5805 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:46:02.913816 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900480 5805 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:46:02.913816 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900482 5805 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:46:02.913816 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900485 5805 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:46:02.914274 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900488 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:46:02.914274 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900491 5805 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:46:02.914274 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900494 5805 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:46:02.914274 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900496 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:46:02.914274 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900499 5805 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:46:02.914274 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900502 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:46:02.914274 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900504 5805 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:46:02.914274 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900507 5805 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:46:02.914274 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900509 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:46:02.914274 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900512 5805 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:46:02.914274 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900515 5805 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:46:02.914274 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900517 5805 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:46:02.914274 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900520 5805 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:46:02.914274 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900522 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:46:02.914274 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900525 5805 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:46:02.914274 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900527 5805 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:46:02.914274 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900530 5805 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:46:02.914274 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900532 5805 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:46:02.914274 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900535 5805 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:46:02.914745 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900538 5805 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:46:02.914745 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900540 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:46:02.914745 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900543 5805 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:46:02.914745 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900545 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:46:02.914745 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900548 5805 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:46:02.914745 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900550 5805 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:46:02.914745 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900555 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:46:02.914745 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900559 5805 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:46:02.914745 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.900561 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:46:02.914745 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.905248 5805 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 17:46:02.914745 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.905263 5805 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 17:46:02.914745 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.905283 5805 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 17:46:02.914745 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.905288 5805 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 17:46:02.914745 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905346 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:46:02.914745 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905351 5805 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:46:02.914745 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905354 5805 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:46:02.914745 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905357 5805 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:46:02.914745 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905360 5805 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:46:02.914745 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905364 5805 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:46:02.915201 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905368 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:46:02.915201 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905371 5805 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:46:02.915201 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905374 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:46:02.915201 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905378 5805 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:46:02.915201 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905380 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:46:02.915201 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905383 5805 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:46:02.915201 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905386 5805 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:46:02.915201 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905388 5805 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:46:02.915201 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905416 5805 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:46:02.915201 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905429 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:46:02.915201 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905432 5805 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:46:02.915201 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905435 5805 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:46:02.915201 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905438 5805 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:46:02.915201 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905441 5805 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:46:02.915201 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905443 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:46:02.915201 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905447 5805 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:46:02.915201 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905449 5805 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:46:02.915201 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905452 5805 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:46:02.915201 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905455 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:46:02.915201 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905458 5805 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:46:02.915697 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905460 5805 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:46:02.915697 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905463 5805 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:46:02.915697 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905466 5805 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:46:02.915697 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905469 5805 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:46:02.915697 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905471 5805 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:46:02.915697 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905474 5805 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:46:02.915697 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905476 5805 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:46:02.915697 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905479 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:46:02.915697 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905482 5805 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:46:02.915697 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905484 5805 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:46:02.915697 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905487 5805 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:46:02.915697 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905489 5805 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:46:02.915697 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905492 5805 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:46:02.915697 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905494 5805 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:46:02.915697 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905497 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:46:02.915697 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905500 5805 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:46:02.915697 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905502 5805 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:46:02.915697 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905505 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:46:02.915697 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905507 5805 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:46:02.915697 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905510 5805 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:46:02.916163 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905513 5805 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:46:02.916163 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905516 5805 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:46:02.916163 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905518 5805 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:46:02.916163 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905521 5805 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:46:02.916163 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905524 5805 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:46:02.916163 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905527 5805 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:46:02.916163 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905530 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:46:02.916163 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905532 5805 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:46:02.916163 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905535 5805 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:46:02.916163 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905538 5805 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:46:02.916163 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905542 5805 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:46:02.916163 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905546 5805 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:46:02.916163 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905549 5805 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:46:02.916163 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905551 5805 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:46:02.916163 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905554 5805 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:46:02.916163 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905557 5805 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:46:02.916163 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905560 5805 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:46:02.916163 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905563 5805 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:46:02.916163 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905565 5805 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:46:02.916807 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905568 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:46:02.916807 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905571 5805 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:46:02.916807 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905573 5805 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:46:02.916807 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905576 5805 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:46:02.916807 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905579 5805 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:46:02.916807 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905581 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:46:02.916807 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905584 5805 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:46:02.916807 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905586 5805 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:46:02.916807 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905589 5805 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:46:02.916807 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905592 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:46:02.916807 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905595 5805 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:46:02.916807 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905597 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:46:02.916807 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905600 5805 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:46:02.916807 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905602 5805 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:46:02.916807 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905606 5805 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:46:02.916807 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905609 5805 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:46:02.916807 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905611 5805 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:46:02.916807 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905614 5805 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:46:02.916807 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905616 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:46:02.917522 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905620 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:46:02.917522 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905622 5805 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:46:02.917522 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905625 5805 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:46:02.917522 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905628 5805 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:46:02.917522 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905631 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:46:02.917522 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905633 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:46:02.917522 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905636 5805 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:46:02.917522 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905638 5805 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:46:02.917522 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905641 5805 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:46:02.917522 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905643 5805 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:46:02.917522 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905646 5805 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:46:02.917522 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905648 5805 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:46:02.917522 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905651 5805 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:46:02.917522 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905653 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:46:02.917522 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905656 5805 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:46:02.917522 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905658 5805 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:46:02.917522 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905661 5805 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:46:02.917522 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905663 5805 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:46:02.917522 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905666 5805 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:46:02.917522 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905669 5805 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:46:02.918275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905671 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:46:02.918275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905674 5805 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:46:02.918275 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.905703 5805 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 17:46:02.918275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905864 5805 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:46:02.918275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905870 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:46:02.918275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905873 5805 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:46:02.918275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905876 5805 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:46:02.918275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905879 5805 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:46:02.918275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905882 5805 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:46:02.918275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905885 5805 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:46:02.918275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905887 5805 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:46:02.918275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905890 5805 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:46:02.918275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905892 5805 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:46:02.918275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905895 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:46:02.918275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905897 5805 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:46:02.918275 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905900 5805 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:46:02.918706 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905903 5805 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:46:02.918706 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905905 5805 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:46:02.918706 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905908 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:46:02.918706 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905910 5805 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:46:02.918706 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905913 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:46:02.918706 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905915 5805 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:46:02.918706 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905918 5805 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:46:02.918706 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905920 5805 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:46:02.918706 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905924 5805 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:46:02.918706 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905928 5805 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:46:02.918706 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905931 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:46:02.918706 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905933 5805 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:46:02.918706 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905936 5805 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:46:02.918706 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905938 5805 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:46:02.918706 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905941 5805 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:46:02.918706 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905943 5805 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:46:02.918706 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905946 5805 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:46:02.918706 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905949 5805 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:46:02.918706 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905951 5805 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:46:02.919230 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905954 5805 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:46:02.919230 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905957 5805 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:46:02.919230 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905959 5805 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:46:02.919230 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905962 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:46:02.919230 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905964 5805 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:46:02.919230 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905967 5805 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:46:02.919230 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905970 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:46:02.919230 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905972 5805 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:46:02.919230 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905975 5805 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:46:02.919230 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905977 5805 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:46:02.919230 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905980 5805 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:46:02.919230 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905983 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:46:02.919230 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905985 5805 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:46:02.919230 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905988 5805 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:46:02.919230 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905990 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:46:02.919230 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905993 5805 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:46:02.919230 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905995 5805 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:46:02.919230 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.905998 5805 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:46:02.919230 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906000 5805 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:46:02.919230 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906003 5805 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:46:02.919743 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906006 5805 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:46:02.919743 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906008 5805 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:46:02.919743 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906011 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:46:02.919743 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906013 5805 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:46:02.919743 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906016 5805 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:46:02.919743 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906018 5805 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:46:02.919743 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906021 5805 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:46:02.919743 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906023 5805 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:46:02.919743 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906026 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:46:02.919743 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906029 5805 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:46:02.919743 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906031 5805 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:46:02.919743 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906034 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:46:02.919743 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906036 5805 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:46:02.919743 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906039 5805 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:46:02.919743 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906041 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:46:02.919743 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906044 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:46:02.919743 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906046 5805 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:46:02.919743 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906049 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:46:02.919743 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906051 5805 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:46:02.919743 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906054 5805 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:46:02.920212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906057 5805 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:46:02.920212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906059 5805 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:46:02.920212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906061 5805 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:46:02.920212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906064 5805 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:46:02.920212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906066 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:46:02.920212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906069 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:46:02.920212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906071 5805 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:46:02.920212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906074 5805 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:46:02.920212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906076 5805 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:46:02.920212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906079 5805 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:46:02.920212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906081 5805 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:46:02.920212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906084 5805 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:46:02.920212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906087 5805 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:46:02.920212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906089 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:46:02.920212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906092 5805 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:46:02.920212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906094 5805 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:46:02.920212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906097 5805 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:46:02.920212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906099 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:46:02.920212 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906102 5805 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:46:02.920821 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906104 5805 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:46:02.920821 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906107 5805 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:46:02.920821 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906110 5805 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:46:02.920821 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906112 5805 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:46:02.920821 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906115 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:46:02.920821 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906117 5805 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:46:02.920821 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906120 5805 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:46:02.920821 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906123 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:46:02.920821 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906125 5805 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:46:02.920821 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906128 5805 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:46:02.920821 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906130 5805 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:46:02.920821 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906133 5805 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:46:02.920821 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906135 5805 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:46:02.920821 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906140 5805 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:46:02.920821 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906144 5805 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:46:02.920821 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.906167 5805 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 17:46:02.921209 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906198 5805 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:46:02.921209 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906202 5805 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:46:02.921209 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906205 5805 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:46:02.921209 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906207 5805 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:46:02.921209 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906210 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:46:02.921209 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906213 5805 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:46:02.921209 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906216 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:46:02.921209 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906218 5805 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:46:02.921209 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906221 5805 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:46:02.921209 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906224 5805 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:46:02.921209 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906227 5805 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:46:02.921209 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906231 5805 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:46:02.921209 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906234 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:46:02.921209 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906245 5805 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:46:02.921209 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906248 5805 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:46:02.921209 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906251 5805 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:46:02.921209 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906253 5805 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:46:02.921209 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906256 5805 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:46:02.921209 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906259 5805 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:46:02.921209 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906261 5805 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:46:02.921698 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906264 5805 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:46:02.921698 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906266 5805 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:46:02.921698 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906269 5805 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:46:02.921698 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906272 5805 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:46:02.921698 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906275 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:46:02.921698 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906277 5805 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:46:02.921698 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906280 5805 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:46:02.921698 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906282 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:46:02.921698 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906285 5805 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:46:02.921698 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906287 5805 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:46:02.921698 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906291 5805 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:46:02.921698 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906294 5805 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:46:02.921698 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906297 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:46:02.921698 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906300 5805 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:46:02.921698 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906302 5805 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:46:02.921698 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906305 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:46:02.921698 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906307 5805 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:46:02.921698 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906310 5805 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:46:02.921698 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906313 5805 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:46:02.921698 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906315 5805 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:46:02.922196 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906318 5805 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:46:02.922196 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906320 5805 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:46:02.922196 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906323 5805 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:46:02.922196 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906325 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:46:02.922196 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906328 5805 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:46:02.922196 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906333 5805 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:46:02.922196 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906336 5805 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:46:02.922196 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906338 5805 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:46:02.922196 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906341 5805 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:46:02.922196 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906343 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:46:02.922196 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906346 5805 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:46:02.922196 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906349 5805 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:46:02.922196 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906351 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:46:02.922196 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906354 5805 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:46:02.922196 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906356 5805 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:46:02.922196 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906359 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:46:02.922196 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906361 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:46:02.922196 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906364 5805 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:46:02.922196 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906366 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:46:02.922196 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906369 5805 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:46:02.922714 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906372 5805 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:46:02.922714 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906374 5805 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:46:02.922714 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906377 5805 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:46:02.922714 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906380 5805 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:46:02.922714 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906383 5805 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:46:02.922714 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906386 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:46:02.922714 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906389 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:46:02.922714 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906406 5805 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:46:02.922714 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906410 5805 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:46:02.922714 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906415 5805 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:46:02.922714 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906417 5805 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:46:02.922714 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906420 5805 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:46:02.922714 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906425 5805 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:46:02.922714 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906428 5805 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:46:02.922714 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906431 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:46:02.922714 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906433 5805 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:46:02.922714 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906436 5805 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:46:02.922714 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906440 5805 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:46:02.922714 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906442 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:46:02.923158 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906445 5805 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:46:02.923158 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906448 5805 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:46:02.923158 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906450 5805 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:46:02.923158 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906453 5805 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:46:02.923158 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906456 5805 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:46:02.923158 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906458 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:46:02.923158 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906461 5805 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:46:02.923158 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906464 5805 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:46:02.923158 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906466 5805 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:46:02.923158 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906469 5805 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:46:02.923158 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906472 5805 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:46:02.923158 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906474 5805 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:46:02.923158 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906477 5805 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:46:02.923158 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906479 5805 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:46:02.923158 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906483 5805 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:46:02.923158 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906487 5805 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:46:02.923158 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906490 5805 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:46:02.923158 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906494 5805 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:46:02.923158 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906496 5805 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:46:02.923705 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906499 5805 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:46:02.923705 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906501 5805 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:46:02.923705 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906505 5805 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:46:02.923705 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906508 5805 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:46:02.923705 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906510 5805 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:46:02.923705 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906513 5805 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:46:02.923705 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906517 5805 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:46:02.923705 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:02.906519 5805 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:46:02.923705 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.906570 5805 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 17:46:02.923705 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.907522 5805 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 17:46:02.923705 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.907581 5805 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 17:46:02.923705 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.907759 5805 server.go:1014] "Starting client certificate rotation" Nov 03 17:46:02.923705 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.907903 5805 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 17:46:02.923705 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.907949 5805 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 17:39:49 +0000 UTC" deadline="2028-08-13 13:44:52.767573638 +0000 UTC" Nov 03 17:46:02.923705 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.907985 5805 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15571h58m49.85959091s" Nov 03 17:46:02.923705 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.908185 5805 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 17:46:02.924096 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.909192 5805 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 17:46:02.924096 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.910288 5805 log.go:25] "Validated CRI v1 runtime API" Nov 03 17:46:02.924096 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.912310 5805 log.go:25] "Validated CRI v1 image API" Nov 03 17:46:02.924096 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.913047 5805 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 17:46:02.924096 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.915266 5805 fs.go:135] Filesystem UUIDs: map[01eaba44-e3ba-432a-9ee3-8de92be6863c:/dev/nvme0n1p3 70390cd0-9b7f-404a-a0b9-f5b6eeed78c0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 03 17:46:02.924246 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.915286 5805 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/160ad685081be00afa353761e3e9725471a84deb4ed3ec1ea7e5f0daeddda5f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/160ad685081be00afa353761e3e9725471a84deb4ed3ec1ea7e5f0daeddda5f0/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1df7518f89cd7da9871f6d8d169e25bd3f80e19ae487165c824c1b3bec821187/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1df7518f89cd7da9871f6d8d169e25bd3f80e19ae487165c824c1b3bec821187/userdata/shm major:0 minor:420 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3031219dfae0399fd53b98cf4d137df1b581cdca1752aa95c95b31c8fd6f715a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3031219dfae0399fd53b98cf4d137df1b581cdca1752aa95c95b31c8fd6f715a/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41b9d83a33d48f80a45e14cb54ed997cb98bc4861f5c04746ad518e117089c82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41b9d83a33d48f80a45e14cb54ed997cb98bc4861f5c04746ad518e117089c82/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4357fa06359bdb104dcf16e83dd04927773998373e48cd06ac23b95f7e7441fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4357fa06359bdb104dcf16e83dd04927773998373e48cd06ac23b95f7e7441fb/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b21927369d3f0d71c349f5d70434c29a3217f3ced7e85f5928a95c267285ab9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b21927369d3f0d71c349f5d70434c29a3217f3ced7e85f5928a95c267285ab9/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d74568c685e0144ec9932191f629f76220da113cbdff3176f508cc6984c07c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d74568c685e0144ec9932191f629f76220da113cbdff3176f508cc6984c07c2/userdata/shm major:0 minor:404 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d8b95e229d388e4131f73e09a14dc0be4f6872d1806b548cafc9ceb8b8e5900/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d8b95e229d388e4131f73e09a14dc0be4f6872d1806b548cafc9ceb8b8e5900/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f96e593da76921ede9472c64a38ccc47f916b3cfa0e24a03ad6c00752045278/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f96e593da76921ede9472c64a38ccc47f916b3cfa0e24a03ad6c00752045278/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7eaa7527d445149c20c06e95d2e090058c2cb7f5f069061435fe92fc52a81e16/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7eaa7527d445149c20c06e95d2e090058c2cb7f5f069061435fe92fc52a81e16/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7eef05cfbc71d8bb1a328dfc16ed154eef1b84b82e1c73a05bc7b8175170dd99/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7eef05cfbc71d8bb1a328dfc16ed154eef1b84b82e1c73a05bc7b8175170dd99/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/813fa771298c166a6343b5651245a1ed32536335e2e63b6e357d6f4210b18c70/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/813fa771298c166a6343b5651245a1ed32536335e2e63b6e357d6f4210b18c70/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8417337ed283c083b428775224f475c51445e5bbc02329e40d863a669796181a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8417337ed283c083b428775224f475c51445e5bbc02329e40d863a669796181a/userdata/shm major:0 minor:310 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/976df117fe7139e172bcc029b789632d77154ac99f193d221dd5b15cf390e9ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/976df117fe7139e172bcc029b789632d77154ac99f193d221dd5b15cf390e9ea/userdata/shm major:0 minor:417 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6ef76c9ffbb6353f61b51f36eec06480fe19c75a27eb2a0bef3dda8b1051114/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6ef76c9ffbb6353f61b51f36eec06480fe19c75a27eb2a0bef3dda8b1051114/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b85290cd780a3d66a3e633e6c560f1ff0ccdb7ee32ac59f151f6cb529ee08235/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b85290cd780a3d66a3e633e6c560f1ff0ccdb7ee32ac59f151f6cb529ee08235/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bac3983211e839f7b74fd7fdbd1e2c9a0f1bcdc526035eb349559a320bda65f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bac3983211e839f7b74fd7fdbd1e2c9a0f1bcdc526035eb349559a320bda65f9/userdata/shm major:0 minor:416 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d84cbd4a9e96f2d71b3b5c106d214bb0de8bea018ee8faa25233073f3fb7df92/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d84cbd4a9e96f2d71b3b5c106d214bb0de8bea018ee8faa25233073f3fb7df92/userdata/shm major:0 minor:256 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7ba28d804f716a25a76352467895bfb7569f8fb85ae1ae38a4c4242cd6a9c9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7ba28d804f716a25a76352467895bfb7569f8fb85ae1ae38a4c4242cd6a9c9b/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e837bd97a0ae1a25f103e74c16ee1958b0b28214340fc5062648f760b906d683/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e837bd97a0ae1a25f103e74c16ee1958b0b28214340fc5062648f760b906d683/userdata/shm major:0 minor:331 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e90efb6b9b1043c03eb07217f59b0009c5403b452caf89e81ff6926af9673b50/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e90efb6b9b1043c03eb07217f59b0009c5403b452caf89e81ff6926af9673b50/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e9773a1686de25e15f4703384ffceb3aa5599c28b1cb8d222263d28cbc30feed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e9773a1686de25e15f4703384ffceb3aa5599c28b1cb8d222263d28cbc30feed/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f6c1cef46b79d828e3a40b0616b5ef026150386d5da980d30f50b9969d0f0e8d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f6c1cef46b79d828e3a40b0616b5ef026150386d5da980d30f50b9969d0f0e8d/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ba1ef12-3f5d-4449-8125-4a1921ec2990/volumes/kubernetes.io~projected/kube-api-access-gznl6:{mountpoint:/var/lib/kubelet/pods/0ba1ef12-3f5d-4449-8125-4a1921ec2990/volumes/kubernetes.io~projected/kube-api-access-gznl6 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ba1ef12-3f5d-4449-8125-4a1921ec2990/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/0ba1ef12-3f5d-4449-8125-4a1921ec2990/volumes/kubernetes.io~secret/metrics-certs major:0 minor:413 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f7aa991-10ec-45ff-ac60-547adf0501a2/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/0f7aa991-10ec-45ff-ac60-547adf0501a2/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f7aa991-10ec-45ff-ac60-547adf0501a2/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/0f7aa991-10ec-45ff-ac60-547adf0501a2/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f7aa991-10ec-45ff-ac60-547adf0501a2/volumes/kubernetes.io~projected/kube-api-access-8jmjl:{mountpoint:/var/lib/kubelet/pods/0f7aa991-10ec-45ff-ac60-547adf0501a2/volumes/kubernetes.io~projected/kube-api-access-8jmjl major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18c90190-94ae-4270-b735-a46d3aaa3f42/volumes/kubernetes.io~projected/kube-api-access-d5flr:{mountpoint:/var/lib/kubelet/pods/18c90190-94ae-4270-b735-a46d3aaa3f42/volumes/kubernetes.io~projected/kube-api-access-d5flr major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18c90190-94ae-4270-b735-a46d3aaa3f42/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/18c90190-94ae-4270-b735-a46d3aaa3f42/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93/volumes/kubernetes.io~projected/kube-api-access-6lcbs:{mountpoint:/var/lib/kubelet/pods/32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93/volumes/kubernetes.io~projected/kube-api-access-6lcbs major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40fbf434-33db-4779-acf2-a148d4cb8c60/volumes/kubernetes.io~projected/kube-api-access-hmhkc:{mountpoint:/var/lib/kubelet/pods/40fbf434-33db-4779-acf2-a148d4cb8c60/volumes/kubernetes.io~projected/kube-api-access-hmhkc major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40fbf434-33db-4779-acf2-a148d4cb8c60/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/40fbf434-33db-4779-acf2-a148d4cb8c60/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189/volumes/kubernetes.io~projected/kube-api-access-ljxz9:{mountpoint:/var/lib/kubelet/pods/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189/volumes/kubernetes.io~projected/kube-api-access-ljxz9 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volume-subpaths/web-config/alertmanager/9 major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volume-subpaths/web-config/config-reloader/6 major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~empty-dir/config-out major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~projected/kube-api-access-hw4rq:{mountpoint:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~projected/kube-api-access-hw4rq major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~projected/tls-assets major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~secret/config-volume major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~secret/web-config major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volume-subpaths/web-config/config-reloader/2 major:0 minor:400 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:400 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volume-subpaths/web-config/prometheus/15 major:0 minor:400 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~empty-dir/config-out major:0 minor:399 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~projected/kube-api-access-xnmr6:{mountpoint:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~projected/kube-api-access-xnmr6 major:0 minor:403 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~projected/tls-assets major:0 minor:395 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/config major:0 minor:401 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:393 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:389 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:396 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:398 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:394 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:402 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:397 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/web-config major:0 minor:400 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62507a50-2bff-4f45-82e7-aca45c032546/volumes/kubernetes.io~projected/kube-api-access-q6dcg:{mountpoint:/var/lib/kubelet/pods/62507a50-2bff-4f45-82e7-aca45c032546/volumes/kubernetes.io~projected/kube-api-access-q6dcg major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/volumes/kubernetes.io~projected/kube-api-access-t2jll:{mountpoint:/var/lib/kubelet/pods/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/volumes/kubernetes.io~projected/kube-api-access-t2jll major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78a8038b-489c-44f5-827f-d4aaebca42f8/volumes/kubernetes.io~projected/kube-api-access-9sm4t:{mountpoint:/var/lib/kubelet/pods/78a8038b-489c-44f5-827f-d4aaebca42f8/volumes/kubernetes.io~projected/kube-api-access-9sm4t major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78a8038b-489c-44f5-827f-d4aaebca42f8/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/78a8038b-489c-44f5-827f-d4aaebca42f8/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78a8038b-489c-44f5-827f-d4aaebca42f8/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/78a8038b-489c-44f5-827f-d4aaebca42f8/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/815e56fd-a11f-4a53-8d85-bfc073f33ea8/volumes/kubernetes.io~projected/kube-api-access-lzmxx:{mountpoint:/var/lib/kubelet/pods/815e56fd-a11f-4a53-8d85-bfc073f33ea8/volumes/kubernetes.io~projected/kube-api-access-lzmxx major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83572d76-31ae-42e0-b7d4-acc08df84a9c/volumes/kubernetes.io~projected/kube-api-access-mnv9j:{mountpoint:/var/lib/kubelet/pods/83572d76-31ae-42e0-b7d4-acc08df84a9c/volumes/kubernetes.io~projected/kube-api-access-mnv9j major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83572d76-31ae-42e0-b7d4-acc08df84a9c/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/83572d76-31ae-42e0-b7d4-acc08df84a9c/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83572d76-31ae-42e0-b7d4-acc08df84a9c/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/83572d76-31ae-42e0-b7d4-acc08df84a9c/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8abced1-c904-4e84-b62e-96290db6fa8e/volumes/kubernetes.io~projected/kube-api-access-tb74w:{mountpoint:/var/lib/kubelet/pods/a8abced1-c904-4e84-b62e-96290db6fa8e/volumes/kubernetes.io~projected/kube-api-access-tb74w major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8abced1-c904-4e84-b62e-96290db6fa8e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a8abced1-c904-4e84-b62e-96290db6fa8e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8abced1-c904-4e84-b62e-96290db6fa8e/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/a8abced1-c904-4e84-b62e-96290db6fa8e/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c527a13f-da23-4228-a048-4af962c36340/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/c527a13f-da23-4228-a048-4af962c36340/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:414 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5890bb8-3e6a-4250-8ccb-50771cd112c0/volumes/kubernetes.io~projected/kube-api-access-l2c2r:{mountpoint:/var/lib/kubelet/pods/c5890bb8-3e6a-4250-8ccb-50771cd112c0/volumes/kubernetes.io~projected/kube-api-access-l2c2r major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5890bb8-3e6a-4250-8ccb-50771cd112c0/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/c5890bb8-3e6a-4250-8ccb-50771cd112c0/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5890bb8-3e6a-4250-8ccb-50771cd112c0/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/c5890bb8-3e6a-4250-8ccb-50771cd112c0/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5890bb8-3e6a-4250-8ccb-50771cd112c0/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/c5890bb8-3e6a-4250-8ccb-50771cd112c0/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5890bb8-3e6a-4250-8ccb-50771cd112c0/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/c5890bb8-3e6a-4250-8ccb-50771cd112c0/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9107231-c3e3-4d76-81e6-b276b828c9bc/volumes/kubernetes.io~projected/kube-api-access-6tv8z:{mountpoint:/var/lib/kubelet/pods/c9107231-c3e3-4d76-81e6-b276b828c9bc/volumes/kubernetes.io~projected/kube-api-access-6tv8z major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2c0ef79-ea05-4061-b4dd-31edac551213/volumes/kubernetes.io~projected/kube-api-access-48n9p:{mountpoint:/var/lib/kubelet/pods/d2c0ef79-ea05-4061-b4dd-31edac551213/volumes/kubernetes.io~projected/kube-api-access-48n9p major:0 minor:415 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3a9f32d-0e80-42ce-81ad-d4c08790fccb/volumes/kubernetes.io~projected/kube-api-access-8997w:{mountpoint:/var/lib/kubelet/pods/d3a9f32d-0e80-42ce-81ad-d4c08790fccb/volumes/kubernetes.io~projected/kube-api-access-8997w major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d82f08c7-9da6-4969-858f-c8e5b140d27c/volumes/kubernetes.io~projected/kube-api-access-v5m9x:{mountpoint:/var/lib/kubelet/pods/d82f08c7-9da6-4969-858f-c8e5b140d27c/volumes/kubernetes.io~projected/kube-api-access-v5m9x major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d82f08c7-9da6-4969-858f-c8e5b140d27c/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/d82f08c7-9da6-4969-858f-c8e5b140d27c/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d82f08c7-9da6-4969-858f-c8e5b140d27c/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/d82f08c7-9da6-4969-858f-c8e5b140d27c/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d82f08c7-9da6-4969-858f-c8e5b140d27c/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/d82f08c7-9da6-4969-858f-c8e5b140d27c/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc524cbd-84c3-4255-a208-26fd55af457e/volumes/kubernetes.io~projected/kube-api-access-kdzrj:{mountpoint:/var/lib/kubelet/pods/dc524cbd-84c3-4255-a208-26fd55af457e/volumes/kubernetes.io~projected/kube-api-access-kdzrj major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa56a595-a9ef-431d-b176-41cb1cf382d7/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/fa56a595-a9ef-431d-b176-41cb1cf382d7/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a063a896cfc15ef841cc09688eeae18c72325cf50175d1d89c47e54d98db88b7/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/086eb9c1fb8d43a96e9fbc0a0f900000d68ebacd82f6054bb901b344c9a4a885/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/89dec1381a1c89f63a3973d3585413d4b4debc424dc2e62c0987be9063d42866/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/d14600d347cd1d79e1c4a8d2d3461eb45d7b4d0d4de9eccf923234a7f57d6d20/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/305676f90b99fa2718ba76e8e67cf6f82e912a9953fc165a819ac4b2feefbfc5/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/360b0852de1d61b7d01b4dc6ec2d17d2701418dc78efb04997c2c3ec4ee859d1/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/cdf3bf7a23bf7bfa4188485971bce1a1c705138c555c677f01dbdababc9dede9/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/be1cefeaf54b2fff404c16eb601cec6449b20f077f7e7ed9d9322b771a2cb568/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/b7d5f978f47707604635741cf454fe52bdd1434ccc2ae73035e4b4b665199ba2/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/b403f13e3ef6da2df302442fac2812726329cbb3e14ec9329b7c6ca1143180d1/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/5c692096bee468e1de1199f5d4cca00f7683bd8bdb7f881c8112b011481b04ea/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/8f0ba0aa7ec7c32b3789b359977d8f28ca730578e09551e52b3bc25a802c9756/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/79c2ac5372019c564eb4cbebeebe6218bfbe0eb98d1ea4ba6d4940ac4ae6df68/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/fb1762e7bcc0a51b264b98b74226d61dd1a863c6890fa94aac7b71ada29f26b5/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/cd0bdbf1b56610fe9be5636560261bb3130b75922b58f70e69b6154284b180ef/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/6efab1274ceac4cd5c3cee409b43a2bff457f3705088d3f63079e73877317382/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/46e0e179d9c0aadd574a6eff9829b24d22f787b4cfb11285bb402ec8ec547f1f/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/ca992798fe8eea774a737d5e8e4b208a99792ff64b546e1e143a3887e4dbe3ae/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/2852c35bf441fb71f1ebd57277670dc9c66bf1cd6a7cda9ed96d0581124b55a6/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/aff335290d7012f5873f7a8a9736c08356a07adee78202458be398787fdb4738/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/27d7d571084dcb6035dccc03a25e33d47b6bedbc9d62439f54fbcd016193a042/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/d93e40874c13211832cfeacb81627cb964c8b930fd4ef4c9557fe4edff54cc2a/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/0edc3a4a03e535575c4b7103fdd6188269b44bc154fc1f69eeda749a83f07ec4/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/4c5a49013938d545d33d32c89e46972fd4220ccab2959606dcbd6e0ba56d4bbc/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/c7785e1797c1c02afc7cb8d236fdd603c24ff19d2c75b685d57c86269179d403/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/62710547f1c647588c28e91a9b26da16e9dda5d3ba57d42a5458800b13cd4b31/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/283314ffa73847aa70962d35e7edd5e9865f60e7171dd251083f0e8c11dec767/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/01c8d72d7ca73de46336d8e712b6f1cc669ea569b2d6f0c196c047dd24664f5d/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/3ce82e1ac78c175c2b23aebe363b7ddc2914859baca27e433c79d51a8086eeb2/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/cf188f1961b17f64de72a68499d4f36b74fc02e5fdb73c6a861ec8bf31be2ea6/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/962c0b91973acd98b036ea0388ec021a5ef79c9f874e98025dd5762116cbe8c4/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/76416b710100d3478d4540ab93a2a2077c6a902e4adedbac2a2ce266b3f0b3c7/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/4804c77e0969d7e2eca4ae5b2001596bc10560d33e928e591fb8e4655f037d6f/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/126cb2eacb8bc51d17fcb87f36f123f99745369ce7696ddc01e7fd584c0822e0/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/7a473a4d739ee2496c157bfa3624ec1adb400cd983ba0da51647abb311ed8df8/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/2272e10152f80da9379694befadfb000bb0f06b413d8db9c28ff2790289cb6ae/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/81c25be78a1f6a602876809d9d0c81b8a71fd099ea24d887c898343b7399225f/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/ab7cd67976ec3fa3d53c4af7704eb98c3b95827b000455b40bae936c63cd8508/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/31984efb0c08f3d553fbe62e862d395c8ca60a3f07722a0abd576e826c7388a0/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/cafc72872222733eea196ecc3bb03de82545d8d28f12026fa5d2325cef957af4/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/ede90cf6a158e10bed22b16ec907a89e6eb26a367de7e3e64098f2b42dc297c7/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/3f09b87a9514df88be4c7ee3bc3e5eb1652ead111c3ba2a7733d93b063143063/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/f9cdc3c783296eee418203be25470474c7671f0e0a0fb9825f765accdd7fe4e9/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/cbeca1d6e0f3e62a16215c8f4eeee067d743a2e8642778b34f19c55fc8114933/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/8685d69e6e2116d53918a24c1e799c1829dd8e34f3552f9eab39e6c7a39998f5/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/9c25672df597ace4ed0e094e9c49f72af752b2aabfff8f0b8b0933570a4f6f9b/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/c93b5e6bd27139ebfae66f601fe5a3d2e052f4949e2f0f3d88fb5825dc4b6dcc/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/4092b6d24029276aa2c5a1b7af1817e2e7623953963f436951b729691f800de2/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/8c6521cd752469aefcc72f30667dc431eb9291d94e6eddcf7a359e33be3c8b4d/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/2d87d7a83705182ce457ecf7eef402afd418b9810a3948ada5247f3c45cbac34/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/24890f60b3c43d8e05fc938c537f262511efa9700d7a568b35c87f7d99d5b01c/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/a7f3b43ae17f6867dedaab8e6f3629a22a34b7c06dd35ffddc36a2500e22e694/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-384:{mountpoint:/var/lib/containers/storage/overlay/6195e9a0bce851ee604f5c3a86cfca07b604484c7585fe95328be5fb0d715dcb/merged major:0 minor:384 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/cd178ee72f98a0f4f81223ade10fee20610bc037818757831c28a522d3ed2fc7/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/0f7101031ff2e3f972a31860bc44e3cce2507c4bb08424103a00c055f50173c6/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-422:{mountpoint:/var/lib/containers/storage/overlay/091df7f4525236375d736a8f3c15e37e6193cf851325e372473a73b70a7907ff/merged major:0 minor:422 fsType:overlay blockSize:0} overlay_0-424:{mountpoint:/var/lib/containers/storage/overlay/ab5a31b1ec4a53bea23ba2394be134edb178b51f700969bbaf0bd9e058243bff/merged major:0 minor:424 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/6663d2fac65bd13542183d8b8c27caabfd95498e62695d7454e3d8387c7c585f/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-428:{mountpoint:/var/lib/containers/storage/overlay/ca001df9a2ef398b11b2ecd83030f0bf4a994c1d1c322dc19e67be6b2f20b832/merged major:0 minor:428 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/28f73190208bcb7c54ed89546ab41cdd8e59a3391e3e807611440ce467a4a322/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/66a740e7abaf21e3f7bb901f820f95c46085e85cd0d44e5e231691c2240e8596/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/015149e4c5bacd63b9f0c5c4f373d57f291387a0e3019df1b8e498a914167514/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/b007e748b39869ffeaa04d6c5f092f4fa4cb929ac2bbb7b7983c11041af96c06/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-452:{mountpoint:/var/lib/containers/storage/overlay/ff151573944d76cfadb87a81e92777f30b3b105d1dba484bd20ebc16132499c8/merged major:0 minor:452 fsType:overlay blockSize:0} overlay_0-458:{mountpoint:/var/lib/containers/storage/overlay/d7d0e3283af21b821c7734573f0fe47a16c78a8c65b9467eeaf0ad08606431b2/merged major:0 minor:458 fsType:overlay blockSize:0} overlay_0-460:{mountpoint:/var/lib/containers/storage/overlay/93e676d3f8ca7b0374c07021e11ecc58209ea3f12922c5ad85e1c38063836bef/merged major:0 minor:460 fsType:overlay blockSize:0} overlay_0-469:{mountpoint:/var/lib/containers/storage/overlay/f74119967f99b16f8871c4dcd693d5d57aa40f1e553401e82a8f6cb4ef1056cb/merged major:0 minor:469 fsType:overlay blockSize:0} overlay_0-474:{mountpoint:/var/lib/containers/storage/overlay/3584b6981b3ce3985a9ef2bb238c2427cf8cb3316f0b4a03bc2f49554fbf4573/merged major:0 minor:474 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/473c387ebe45da3ebd763c614ad1b27a9bc0202046ee0078041e9268ec552cc2/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/e63f2fb72839fa27644160637947ee6c72ef43604efff80c5cb42f368299ed7d/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/f748054db782ccb8962b28cccf231cedc2435f99e4dab3f054be2c25b85114dc/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/f0345096c3a651f45e24ee5e90c9904647e83163b081ed56cf35a23318684a7e/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/b01ccbd1709a056c22a2e659d54c06d54f14426138e07a99d221accd4cd2e062/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/3db0de5ce56bea4ccb62b87028569d887c8eb8803884be410a152cf85b04f03b/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/9c71a17910c01e9420218d6901731c56d6b364eedf0bb98035b252a62074851e/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/97f79d275869c70ecce8353978147e9ea35e76f922578d6ede4095adc1da48cb/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/4dd1cf7255c88122d9193ba52b3865ccabf4ff9e1d21989220ed8e7630376af5/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/687792f3f6bb0d13543d85447fa85b7b43ff355de90920ed35c6514ea883bd17/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 17:46:02.924558 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.923575 5805 manager.go:217] Machine: {Timestamp:2026-11-03 17:46:02.923099949 +0000 UTC m=+0.056728950 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097289 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22447227635aa58c11e2e0dbb1a41e SystemUUID:ec224472-2763-5aa5-8c11-e2e0dbb1a41e BootID:24add276-db74-4d25-aab0-703b065e9c43 Filesystems:[{Device:/var/lib/kubelet/pods/0f7aa991-10ec-45ff-ac60-547adf0501a2/volumes/kubernetes.io~projected/kube-api-access-8jmjl DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:291 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8417337ed283c083b428775224f475c51445e5bbc02329e40d863a669796181a/userdata/shm DeviceMajor:0 DeviceMinor:310 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d82f08c7-9da6-4969-858f-c8e5b140d27c/volumes/kubernetes.io~projected/kube-api-access-v5m9x DeviceMajor:0 DeviceMinor:330 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:298 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:296 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d8b95e229d388e4131f73e09a14dc0be4f6872d1806b548cafc9ceb8b8e5900/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8abced1-c904-4e84-b62e-96290db6fa8e/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:258 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:400 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d84cbd4a9e96f2d71b3b5c106d214bb0de8bea018ee8faa25233073f3fb7df92/userdata/shm DeviceMajor:0 DeviceMinor:256 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:294 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c5890bb8-3e6a-4250-8ccb-50771cd112c0/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:336 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:389 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:398 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:400 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/160ad685081be00afa353761e3e9725471a84deb4ed3ec1ea7e5f0daeddda5f0/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0ba1ef12-3f5d-4449-8125-4a1921ec2990/volumes/kubernetes.io~projected/kube-api-access-gznl6 DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41b9d83a33d48f80a45e14cb54ed997cb98bc4861f5c04746ad518e117089c82/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40fbf434-33db-4779-acf2-a148d4cb8c60/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40fbf434-33db-4779-acf2-a148d4cb8c60/volumes/kubernetes.io~projected/kube-api-access-hmhkc DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:291 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~projected/kube-api-access-hw4rq DeviceMajor:0 DeviceMinor:304 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c5890bb8-3e6a-4250-8ccb-50771cd112c0/volumes/kubernetes.io~projected/kube-api-access-l2c2r DeviceMajor:0 DeviceMinor:339 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3031219dfae0399fd53b98cf4d137df1b581cdca1752aa95c95b31c8fd6f715a/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:396 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d74568c685e0144ec9932191f629f76220da113cbdff3176f508cc6984c07c2/userdata/shm DeviceMajor:0 DeviceMinor:404 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-422 DeviceMajor:0 DeviceMinor:422 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7eef05cfbc71d8bb1a328dfc16ed154eef1b84b82e1c73a05bc7b8175170dd99/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:296 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:402 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-424 DeviceMajor:0 DeviceMinor:424 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fa56a595-a9ef-431d-b176-41cb1cf382d7/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f96e593da76921ede9472c64a38ccc47f916b3cfa0e24a03ad6c00752045278/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a8abced1-c904-4e84-b62e-96290db6fa8e/volumes/kubernetes.io~projected/kube-api-access-tb74w DeviceMajor:0 DeviceMinor:260 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c5890bb8-3e6a-4250-8ccb-50771cd112c0/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:338 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:400 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-452 DeviceMajor:0 DeviceMinor:452 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4357fa06359bdb104dcf16e83dd04927773998373e48cd06ac23b95f7e7441fb/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:295 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:394 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:399 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:401 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-474 DeviceMajor:0 DeviceMinor:474 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f7aa991-10ec-45ff-ac60-547adf0501a2/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7ba28d804f716a25a76352467895bfb7569f8fb85ae1ae38a4c4242cd6a9c9b/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18c90190-94ae-4270-b735-a46d3aaa3f42/volumes/kubernetes.io~projected/kube-api-access-d5flr DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1df7518f89cd7da9871f6d8d169e25bd3f80e19ae487165c824c1b3bec821187/userdata/shm DeviceMajor:0 DeviceMinor:420 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c9107231-c3e3-4d76-81e6-b276b828c9bc/volumes/kubernetes.io~projected/kube-api-access-6tv8z DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c5890bb8-3e6a-4250-8ccb-50771cd112c0/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:337 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c527a13f-da23-4228-a048-4af962c36340/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:414 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-428 DeviceMajor:0 DeviceMinor:428 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c5890bb8-3e6a-4250-8ccb-50771cd112c0/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:335 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78a8038b-489c-44f5-827f-d4aaebca42f8/volumes/kubernetes.io~projected/kube-api-access-9sm4t DeviceMajor:0 DeviceMinor:252 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:393 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d2c0ef79-ea05-4061-b4dd-31edac551213/volumes/kubernetes.io~projected/kube-api-access-48n9p DeviceMajor:0 DeviceMinor:415 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/62507a50-2bff-4f45-82e7-aca45c032546/volumes/kubernetes.io~projected/kube-api-access-q6dcg DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93/volumes/kubernetes.io~projected/kube-api-access-6lcbs DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78a8038b-489c-44f5-827f-d4aaebca42f8/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:247 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83572d76-31ae-42e0-b7d4-acc08df84a9c/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:254 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-458 DeviceMajor:0 DeviceMinor:458 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/813fa771298c166a6343b5651245a1ed32536335e2e63b6e357d6f4210b18c70/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f6c1cef46b79d828e3a40b0616b5ef026150386d5da980d30f50b9969d0f0e8d/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dc524cbd-84c3-4255-a208-26fd55af457e/volumes/kubernetes.io~projected/kube-api-access-kdzrj DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189/volumes/kubernetes.io~projected/kube-api-access-ljxz9 DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b85290cd780a3d66a3e633e6c560f1ff0ccdb7ee32ac59f151f6cb529ee08235/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e837bd97a0ae1a25f103e74c16ee1958b0b28214340fc5062648f760b906d683/userdata/shm DeviceMajor:0 DeviceMinor:331 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-384 DeviceMajor:0 DeviceMinor:384 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e9773a1686de25e15f4703384ffceb3aa5599c28b1cb8d222263d28cbc30feed/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d82f08c7-9da6-4969-858f-c8e5b140d27c/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:327 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:395 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:397 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/815e56fd-a11f-4a53-8d85-bfc073f33ea8/volumes/kubernetes.io~projected/kube-api-access-lzmxx DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d3a9f32d-0e80-42ce-81ad-d4c08790fccb/volumes/kubernetes.io~projected/kube-api-access-8997w DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b21927369d3f0d71c349f5d70434c29a3217f3ced7e85f5928a95c267285ab9/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes/kubernetes.io~projected/kube-api-access-xnmr6 DeviceMajor:0 DeviceMinor:403 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/976df117fe7139e172bcc029b789632d77154ac99f193d221dd5b15cf390e9ea/userdata/shm DeviceMajor:0 DeviceMinor:417 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0ba1ef12-3f5d-4449-8125-4a1921ec2990/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:413 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bac3983211e839f7b74fd7fdbd1e2c9a0f1bcdc526035eb349559a320bda65f9/userdata/shm DeviceMajor:0 DeviceMinor:416 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-469 DeviceMajor:0 DeviceMinor:469 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8abced1-c904-4e84-b62e-96290db6fa8e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:259 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e90efb6b9b1043c03eb07217f59b0009c5403b452caf89e81ff6926af9673b50/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-460 DeviceMajor:0 DeviceMinor:460 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0f7aa991-10ec-45ff-ac60-547adf0501a2/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18c90190-94ae-4270-b735-a46d3aaa3f42/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d82f08c7-9da6-4969-858f-c8e5b140d27c/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:329 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:296 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/volumes/kubernetes.io~projected/kube-api-access-t2jll DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/78a8038b-489c-44f5-827f-d4aaebca42f8/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:251 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/83572d76-31ae-42e0-b7d4-acc08df84a9c/volumes/kubernetes.io~projected/kube-api-access-mnv9j DeviceMajor:0 DeviceMinor:255 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d82f08c7-9da6-4969-858f-c8e5b140d27c/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:328 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:400 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6ef76c9ffbb6353f61b51f36eec06480fe19c75a27eb2a0bef3dda8b1051114/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/83572d76-31ae-42e0-b7d4-acc08df84a9c/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:253 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7eaa7527d445149c20c06e95d2e090058c2cb7f5f069061435fe92fc52a81e16/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:297 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:296 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1df7518f89cd7da MacAddress:7e:f7:41:ce:46:e0 Speed:10000 Mtu:8901} {Name:4d74568c685e014 MacAddress:02:64:97:4f:a0:7f Speed:10000 Mtu:8901} {Name:7eaa7527d445149 MacAddress:c2:e3:65:2b:01:6c Speed:10000 Mtu:8901} {Name:7eef05cfbc71d8b MacAddress:2a:5d:c7:c0:57:5f Speed:10000 Mtu:8901} {Name:8417337ed283c08 MacAddress:e6:d1:f9:4d:fb:c0 Speed:10000 Mtu:8901} {Name:976df117fe7139e MacAddress:16:57:c9:e2:29:fe Speed:10000 Mtu:8901} {Name:b85290cd780a3d6 MacAddress:be:b2:0b:3a:34:af Speed:10000 Mtu:8901} {Name:bac3983211e839f MacAddress:b6:e1:fe:b3:d0:8b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:cb:28:2a:e1 Speed:0 Mtu:9001} {Name:br-int MacAddress:be:85:ea:46:d2:dc Speed:0 Mtu:8901} {Name:d84cbd4a9e96f2d MacAddress:da:7f:d9:95:76:4c Speed:10000 Mtu:8901} {Name:e837bd97a0ae1a2 MacAddress:a2:16:14:af:5b:06 Speed:10000 Mtu:8901} {Name:e90efb6b9b1043c MacAddress:2a:97:da:b2:a0:74 Speed:10000 Mtu:8901} {Name:e9773a1686de25e MacAddress:de:27:03:a6:46:48 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:cb:28:2a:e1 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:e6:bf:a1:03:2a:ea Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:a6:5a:68:9e:e3:4f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 17:46:02.924558 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.924031 5805 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 17:46:02.924558 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.924075 5805 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 17:46:02.924558 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.924358 5805 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 17:46:02.924558 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.924379 5805 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-164.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 17:46:02.924558 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.924513 5805 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 17:46:02.924558 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.924520 5805 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 17:46:02.924558 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.924530 5805 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 17:46:02.924558 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.924543 5805 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 17:46:02.924976 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.924616 5805 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 17:46:02.924976 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.924759 5805 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 17:46:02.924976 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.924860 5805 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 17:46:02.924976 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.924892 5805 kubelet.go:493] "Attempting to sync node with API server" Nov 03 17:46:02.924976 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.924901 5805 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 17:46:02.924976 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.924910 5805 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 17:46:02.924976 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.924919 5805 kubelet.go:405] "Adding apiserver pod source" Nov 03 17:46:02.924976 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.924927 5805 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 17:46:02.925799 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.925786 5805 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 17:46:02.925799 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.925800 5805 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 17:46:02.926278 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.926261 5805 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 17:46:02.929145 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.929131 5805 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 17:46:02.929145 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.929145 5805 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 17:46:02.929254 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.929245 5805 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 17:46:02.929455 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.929432 5805 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 17:46:02.929499 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.929462 5805 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 17:46:02.929570 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.929562 5805 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 17:46:02.929603 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.929576 5805 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 17:46:02.929603 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.929582 5805 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 17:46:02.929603 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.929588 5805 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 17:46:02.929603 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.929596 5805 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 17:46:02.929705 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.929607 5805 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 17:46:02.929705 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.929615 5805 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 17:46:02.929705 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.929620 5805 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 17:46:02.929705 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.929628 5805 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 17:46:02.929705 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.929634 5805 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 17:46:02.929705 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.929648 5805 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 17:46:02.929705 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.929658 5805 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 17:46:02.929705 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.929682 5805 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 17:46:02.929705 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.929690 5805 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 17:46:02.933024 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.933009 5805 server.go:1263] "Started kubelet" Nov 03 17:46:02.933150 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.933098 5805 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 17:46:02.933206 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.933177 5805 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 17:46:02.933955 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.933925 5805 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 17:46:02.933955 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.933926 5805 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 17:46:02.936049 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.935624 5805 server.go:317] "Adding debug handlers to kubelet server" Nov 03 17:46:02.934206 ip-10-0-137-164 systemd[1]: Started Kubernetes Kubelet. Nov 03 17:46:02.940527 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.939315 5805 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 17:46:02.940527 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.939333 5805 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 17:46:02.940527 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.939372 5805 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 17:39:51 +0000 UTC" deadline="2028-04-17 13:40:30.149035287 +0000 UTC" Nov 03 17:46:02.940527 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.939418 5805 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12739h54m27.209646471s" Nov 03 17:46:02.940527 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.939446 5805 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 17:46:02.940527 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.939462 5805 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 17:46:02.940527 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:02.939451 5805 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-164.us-west-2.compute.internal\" not found" Nov 03 17:46:02.940527 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.939581 5805 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 17:46:02.941671 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.941648 5805 factory.go:55] Registering systemd factory Nov 03 17:46:02.941756 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.941674 5805 factory.go:223] Registration of the systemd container factory successfully Nov 03 17:46:02.942613 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942571 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ba1ef12-3f5d-4449-8125-4a1921ec2990" volumeName="kubernetes.io/projected/0ba1ef12-3f5d-4449-8125-4a1921ec2990-kube-api-access-gznl6" seLinuxMountContext="" Nov 03 17:46:02.942687 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942619 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e2b3783-58ff-4bb7-b713-6da60023f5ef" volumeName="kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 03 17:46:02.942687 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942634 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="604769b0-503d-444f-983e-265898035e90" volumeName="kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 03 17:46:02.942687 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942646 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="604769b0-503d-444f-983e-265898035e90" volumeName="kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 03 17:46:02.942687 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942658 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a74bba5-bc9c-4ae1-9754-8887c5bf23cc" volumeName="kubernetes.io/secret/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 17:46:02.942687 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942671 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8abced1-c904-4e84-b62e-96290db6fa8e" volumeName="kubernetes.io/projected/a8abced1-c904-4e84-b62e-96290db6fa8e-kube-api-access-tb74w" seLinuxMountContext="" Nov 03 17:46:02.942687 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942683 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8abced1-c904-4e84-b62e-96290db6fa8e" volumeName="kubernetes.io/secret/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-tls" seLinuxMountContext="" Nov 03 17:46:02.942943 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942695 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18c90190-94ae-4270-b735-a46d3aaa3f42" volumeName="kubernetes.io/empty-dir/18c90190-94ae-4270-b735-a46d3aaa3f42-data-volume" seLinuxMountContext="" Nov 03 17:46:02.942943 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942709 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18c90190-94ae-4270-b735-a46d3aaa3f42" volumeName="kubernetes.io/secret/18c90190-94ae-4270-b735-a46d3aaa3f42-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 17:46:02.942943 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942722 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e2b3783-58ff-4bb7-b713-6da60023f5ef" volumeName="kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-cluster-tls-config" seLinuxMountContext="" Nov 03 17:46:02.942943 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942734 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="604769b0-503d-444f-983e-265898035e90" volumeName="kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-metrics-client-ca" seLinuxMountContext="" Nov 03 17:46:02.942943 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942746 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="604769b0-503d-444f-983e-265898035e90" volumeName="kubernetes.io/projected/604769b0-503d-444f-983e-265898035e90-tls-assets" seLinuxMountContext="" Nov 03 17:46:02.942943 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942757 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83572d76-31ae-42e0-b7d4-acc08df84a9c" volumeName="kubernetes.io/secret/83572d76-31ae-42e0-b7d4-acc08df84a9c-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 17:46:02.942943 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942768 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d82f08c7-9da6-4969-858f-c8e5b140d27c" volumeName="kubernetes.io/secret/d82f08c7-9da6-4969-858f-c8e5b140d27c-client-ca-bundle" seLinuxMountContext="" Nov 03 17:46:02.942943 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942791 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc524cbd-84c3-4255-a208-26fd55af457e" volumeName="kubernetes.io/configmap/dc524cbd-84c3-4255-a208-26fd55af457e-iptables-alerter-script" seLinuxMountContext="" Nov 03 17:46:02.942943 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942835 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="604769b0-503d-444f-983e-265898035e90" volumeName="kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 03 17:46:02.942943 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942849 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62507a50-2bff-4f45-82e7-aca45c032546" volumeName="kubernetes.io/configmap/62507a50-2bff-4f45-82e7-aca45c032546-multus-daemon-config" seLinuxMountContext="" Nov 03 17:46:02.942943 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942863 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78a8038b-489c-44f5-827f-d4aaebca42f8" volumeName="kubernetes.io/secret/78a8038b-489c-44f5-827f-d4aaebca42f8-openshift-state-metrics-tls" seLinuxMountContext="" Nov 03 17:46:02.942943 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942874 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5890bb8-3e6a-4250-8ccb-50771cd112c0" volumeName="kubernetes.io/projected/c5890bb8-3e6a-4250-8ccb-50771cd112c0-kube-api-access-l2c2r" seLinuxMountContext="" Nov 03 17:46:02.942943 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942886 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5890bb8-3e6a-4250-8ccb-50771cd112c0" volumeName="kubernetes.io/secret/c5890bb8-3e6a-4250-8ccb-50771cd112c0-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 17:46:02.942943 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942897 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5890bb8-3e6a-4250-8ccb-50771cd112c0" volumeName="kubernetes.io/secret/c5890bb8-3e6a-4250-8ccb-50771cd112c0-telemeter-client-tls" seLinuxMountContext="" Nov 03 17:46:02.942943 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942909 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9107231-c3e3-4d76-81e6-b276b828c9bc" volumeName="kubernetes.io/projected/c9107231-c3e3-4d76-81e6-b276b828c9bc-kube-api-access-6tv8z" seLinuxMountContext="" Nov 03 17:46:02.942943 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942921 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2c0ef79-ea05-4061-b4dd-31edac551213" volumeName="kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p" seLinuxMountContext="" Nov 03 17:46:02.942943 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942933 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18c90190-94ae-4270-b735-a46d3aaa3f42" volumeName="kubernetes.io/projected/18c90190-94ae-4270-b735-a46d3aaa3f42-kube-api-access-d5flr" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942952 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e2b3783-58ff-4bb7-b713-6da60023f5ef" volumeName="kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942965 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="604769b0-503d-444f-983e-265898035e90" volumeName="kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942977 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8abced1-c904-4e84-b62e-96290db6fa8e" volumeName="kubernetes.io/empty-dir/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-textfile" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.942989 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5890bb8-3e6a-4250-8ccb-50771cd112c0" volumeName="kubernetes.io/secret/c5890bb8-3e6a-4250-8ccb-50771cd112c0-federate-client-tls" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943007 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3a9f32d-0e80-42ce-81ad-d4c08790fccb" volumeName="kubernetes.io/configmap/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943019 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d82f08c7-9da6-4969-858f-c8e5b140d27c" volumeName="kubernetes.io/configmap/d82f08c7-9da6-4969-858f-c8e5b140d27c-metrics-server-audit-profiles" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943030 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d82f08c7-9da6-4969-858f-c8e5b140d27c" volumeName="kubernetes.io/secret/d82f08c7-9da6-4969-858f-c8e5b140d27c-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943055 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40fbf434-33db-4779-acf2-a148d4cb8c60" volumeName="kubernetes.io/secret/40fbf434-33db-4779-acf2-a148d4cb8c60-cert" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943067 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a74bba5-bc9c-4ae1-9754-8887c5bf23cc" volumeName="kubernetes.io/configmap/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-ovnkube-script-lib" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943080 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83572d76-31ae-42e0-b7d4-acc08df84a9c" volumeName="kubernetes.io/projected/83572d76-31ae-42e0-b7d4-acc08df84a9c-kube-api-access-mnv9j" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943092 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5890bb8-3e6a-4250-8ccb-50771cd112c0" volumeName="kubernetes.io/secret/c5890bb8-3e6a-4250-8ccb-50771cd112c0-secret-telemeter-client" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943104 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f7aa991-10ec-45ff-ac60-547adf0501a2" volumeName="kubernetes.io/empty-dir/0f7aa991-10ec-45ff-ac60-547adf0501a2-tmp" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943116 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e2b3783-58ff-4bb7-b713-6da60023f5ef" volumeName="kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943128 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="604769b0-503d-444f-983e-265898035e90" volumeName="kubernetes.io/projected/604769b0-503d-444f-983e-265898035e90-kube-api-access-xnmr6" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943140 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c527a13f-da23-4228-a048-4af962c36340" volumeName="kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943152 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d82f08c7-9da6-4969-858f-c8e5b140d27c" volumeName="kubernetes.io/empty-dir/d82f08c7-9da6-4969-858f-c8e5b140d27c-audit-log" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943163 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc524cbd-84c3-4255-a208-26fd55af457e" volumeName="kubernetes.io/projected/dc524cbd-84c3-4255-a208-26fd55af457e-kube-api-access-kdzrj" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943174 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa56a595-a9ef-431d-b176-41cb1cf382d7" volumeName="kubernetes.io/configmap/fa56a595-a9ef-431d-b176-41cb1cf382d7-konnectivity-ca" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943185 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa56a595-a9ef-431d-b176-41cb1cf382d7" volumeName="kubernetes.io/secret/fa56a595-a9ef-431d-b176-41cb1cf382d7-agent-certs" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943196 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f7aa991-10ec-45ff-ac60-547adf0501a2" volumeName="kubernetes.io/empty-dir/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-tuned" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943207 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="604769b0-503d-444f-983e-265898035e90" volumeName="kubernetes.io/empty-dir/604769b0-503d-444f-983e-265898035e90-config-out" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943218 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a74bba5-bc9c-4ae1-9754-8887c5bf23cc" volumeName="kubernetes.io/projected/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-kube-api-access-t2jll" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943229 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ba1ef12-3f5d-4449-8125-4a1921ec2990" volumeName="kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943240 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e2b3783-58ff-4bb7-b713-6da60023f5ef" volumeName="kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-config-volume" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943251 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="604769b0-503d-444f-983e-265898035e90" volumeName="kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943263 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78a8038b-489c-44f5-827f-d4aaebca42f8" volumeName="kubernetes.io/configmap/78a8038b-489c-44f5-827f-d4aaebca42f8-metrics-client-ca" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943274 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83572d76-31ae-42e0-b7d4-acc08df84a9c" volumeName="kubernetes.io/configmap/83572d76-31ae-42e0-b7d4-acc08df84a9c-metrics-client-ca" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943285 5805 factory.go:153] Registering CRI-O factory Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943299 5805 factory.go:223] Registration of the crio container factory successfully Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943287 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="604769b0-503d-444f-983e-265898035e90" volumeName="kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-web-config" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943386 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189" volumeName="kubernetes.io/configmap/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189-config-volume" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943438 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e2b3783-58ff-4bb7-b713-6da60023f5ef" volumeName="kubernetes.io/configmap/4e2b3783-58ff-4bb7-b713-6da60023f5ef-metrics-client-ca" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943450 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e2b3783-58ff-4bb7-b713-6da60023f5ef" volumeName="kubernetes.io/empty-dir/4e2b3783-58ff-4bb7-b713-6da60023f5ef-config-out" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943463 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="604769b0-503d-444f-983e-265898035e90" volumeName="kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943485 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e2b3783-58ff-4bb7-b713-6da60023f5ef" volumeName="kubernetes.io/projected/4e2b3783-58ff-4bb7-b713-6da60023f5ef-tls-assets" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943497 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="604769b0-503d-444f-983e-265898035e90" volumeName="kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943510 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="604769b0-503d-444f-983e-265898035e90" volumeName="kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943357 5805 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943524 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a74bba5-bc9c-4ae1-9754-8887c5bf23cc" volumeName="kubernetes.io/configmap/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-env-overrides" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943537 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93" volumeName="kubernetes.io/projected/32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93-kube-api-access-6lcbs" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943548 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e2b3783-58ff-4bb7-b713-6da60023f5ef" volumeName="kubernetes.io/empty-dir/4e2b3783-58ff-4bb7-b713-6da60023f5ef-alertmanager-main-db" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943552 5805 factory.go:103] Registering Raw factory Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943559 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="604769b0-503d-444f-983e-265898035e90" volumeName="kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943570 5805 manager.go:1199] Started watching for new ooms in manager Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943570 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62507a50-2bff-4f45-82e7-aca45c032546" volumeName="kubernetes.io/projected/62507a50-2bff-4f45-82e7-aca45c032546-kube-api-access-q6dcg" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943626 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="815e56fd-a11f-4a53-8d85-bfc073f33ea8" volumeName="kubernetes.io/projected/815e56fd-a11f-4a53-8d85-bfc073f33ea8-kube-api-access-lzmxx" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943648 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83572d76-31ae-42e0-b7d4-acc08df84a9c" volumeName="kubernetes.io/configmap/83572d76-31ae-42e0-b7d4-acc08df84a9c-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943662 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8abced1-c904-4e84-b62e-96290db6fa8e" volumeName="kubernetes.io/configmap/a8abced1-c904-4e84-b62e-96290db6fa8e-metrics-client-ca" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943675 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5890bb8-3e6a-4250-8ccb-50771cd112c0" volumeName="kubernetes.io/configmap/c5890bb8-3e6a-4250-8ccb-50771cd112c0-metrics-client-ca" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943687 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="604769b0-503d-444f-983e-265898035e90" volumeName="kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943699 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83572d76-31ae-42e0-b7d4-acc08df84a9c" volumeName="kubernetes.io/secret/83572d76-31ae-42e0-b7d4-acc08df84a9c-kube-state-metrics-tls" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943710 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8abced1-c904-4e84-b62e-96290db6fa8e" volumeName="kubernetes.io/secret/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943724 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5890bb8-3e6a-4250-8ccb-50771cd112c0" volumeName="kubernetes.io/configmap/c5890bb8-3e6a-4250-8ccb-50771cd112c0-serving-certs-ca-bundle" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943735 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5890bb8-3e6a-4250-8ccb-50771cd112c0" volumeName="kubernetes.io/configmap/c5890bb8-3e6a-4250-8ccb-50771cd112c0-telemeter-trusted-ca-bundle" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943748 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3a9f32d-0e80-42ce-81ad-d4c08790fccb" volumeName="kubernetes.io/projected/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-kube-api-access-8997w" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943761 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d82f08c7-9da6-4969-858f-c8e5b140d27c" volumeName="kubernetes.io/secret/d82f08c7-9da6-4969-858f-c8e5b140d27c-secret-metrics-server-tls" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943772 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="604769b0-503d-444f-983e-265898035e90" volumeName="kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943785 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a74bba5-bc9c-4ae1-9754-8887c5bf23cc" volumeName="kubernetes.io/configmap/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-ovnkube-config" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943801 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d82f08c7-9da6-4969-858f-c8e5b140d27c" volumeName="kubernetes.io/projected/d82f08c7-9da6-4969-858f-c8e5b140d27c-kube-api-access-v5m9x" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943814 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="604769b0-503d-444f-983e-265898035e90" volumeName="kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-grpc-tls" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943825 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="604769b0-503d-444f-983e-265898035e90" volumeName="kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-config" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943837 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18c90190-94ae-4270-b735-a46d3aaa3f42" volumeName="kubernetes.io/configmap/18c90190-94ae-4270-b735-a46d3aaa3f42-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943849 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189" volumeName="kubernetes.io/empty-dir/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189-tmp-dir" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943860 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189" volumeName="kubernetes.io/projected/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189-kube-api-access-ljxz9" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943874 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e2b3783-58ff-4bb7-b713-6da60023f5ef" volumeName="kubernetes.io/configmap/4e2b3783-58ff-4bb7-b713-6da60023f5ef-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943888 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e2b3783-58ff-4bb7-b713-6da60023f5ef" volumeName="kubernetes.io/projected/4e2b3783-58ff-4bb7-b713-6da60023f5ef-kube-api-access-hw4rq" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943900 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e2b3783-58ff-4bb7-b713-6da60023f5ef" volumeName="kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-web-config" seLinuxMountContext="" Nov 03 17:46:02.943908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943910 5805 manager.go:319] Starting recovery of all containers Nov 03 17:46:02.945753 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.943912 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f7aa991-10ec-45ff-ac60-547adf0501a2" volumeName="kubernetes.io/projected/0f7aa991-10ec-45ff-ac60-547adf0501a2-kube-api-access-8jmjl" seLinuxMountContext="" Nov 03 17:46:02.945853 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.945768 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93" volumeName="kubernetes.io/empty-dir/32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93-tmp-dir" seLinuxMountContext="" Nov 03 17:46:02.945853 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.945791 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189" volumeName="kubernetes.io/secret/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189-metrics-tls" seLinuxMountContext="" Nov 03 17:46:02.945853 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.945806 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e2b3783-58ff-4bb7-b713-6da60023f5ef" volumeName="kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 03 17:46:02.945853 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.945819 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="604769b0-503d-444f-983e-265898035e90" volumeName="kubernetes.io/empty-dir/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-db" seLinuxMountContext="" Nov 03 17:46:02.945853 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.945831 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="604769b0-503d-444f-983e-265898035e90" volumeName="kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-metrics-client-certs" seLinuxMountContext="" Nov 03 17:46:02.945853 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.945844 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78a8038b-489c-44f5-827f-d4aaebca42f8" volumeName="kubernetes.io/secret/78a8038b-489c-44f5-827f-d4aaebca42f8-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 17:46:02.946558 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.945859 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83572d76-31ae-42e0-b7d4-acc08df84a9c" volumeName="kubernetes.io/empty-dir/83572d76-31ae-42e0-b7d4-acc08df84a9c-volume-directive-shadow" seLinuxMountContext="" Nov 03 17:46:02.946558 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.945870 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40fbf434-33db-4779-acf2-a148d4cb8c60" volumeName="kubernetes.io/projected/40fbf434-33db-4779-acf2-a148d4cb8c60-kube-api-access-hmhkc" seLinuxMountContext="" Nov 03 17:46:02.946558 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.945883 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78a8038b-489c-44f5-827f-d4aaebca42f8" volumeName="kubernetes.io/projected/78a8038b-489c-44f5-827f-d4aaebca42f8-kube-api-access-9sm4t" seLinuxMountContext="" Nov 03 17:46:02.946558 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.945895 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8abced1-c904-4e84-b62e-96290db6fa8e" volumeName="kubernetes.io/configmap/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 03 17:46:02.946558 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.945907 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9107231-c3e3-4d76-81e6-b276b828c9bc" volumeName="kubernetes.io/configmap/c9107231-c3e3-4d76-81e6-b276b828c9bc-serviceca" seLinuxMountContext="" Nov 03 17:46:02.946558 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.945918 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3a9f32d-0e80-42ce-81ad-d4c08790fccb" volumeName="kubernetes.io/configmap/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 17:46:02.946558 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.945932 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3a9f32d-0e80-42ce-81ad-d4c08790fccb" volumeName="kubernetes.io/configmap/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-cni-binary-copy" seLinuxMountContext="" Nov 03 17:46:02.946558 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.945944 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d82f08c7-9da6-4969-858f-c8e5b140d27c" volumeName="kubernetes.io/configmap/d82f08c7-9da6-4969-858f-c8e5b140d27c-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 17:46:02.946558 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.945957 5805 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62507a50-2bff-4f45-82e7-aca45c032546" volumeName="kubernetes.io/configmap/62507a50-2bff-4f45-82e7-aca45c032546-cni-binary-copy" seLinuxMountContext="" Nov 03 17:46:02.946558 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.945967 5805 reconstruct.go:97] "Volume reconstruction finished" Nov 03 17:46:02.946558 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.945974 5805 reconciler.go:29] "Reconciler: start to sync state" Nov 03 17:46:02.966817 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.966783 5805 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 17:46:02.968380 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.968352 5805 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 17:46:02.968380 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.968374 5805 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 17:46:02.968550 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.968421 5805 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 17:46:02.968550 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:02.968462 5805 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 17:46:02.985026 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.985005 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 17:46:02.985169 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.985152 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 17:46:02.987140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.987117 5805 generic.go:356] "Generic (PLEG): container finished" podID="604769b0-503d-444f-983e-265898035e90" containerID="15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2" exitCode=0 Nov 03 17:46:02.993811 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.993787 5805 generic.go:356] "Generic (PLEG): container finished" podID="d3a9f32d-0e80-42ce-81ad-d4c08790fccb" containerID="f4e07f7d3464bcdbbd319d25d4fed6106f0685beca6c3cff55cfdaf96582e1c2" exitCode=0 Nov 03 17:46:02.993811 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.993811 5805 generic.go:356] "Generic (PLEG): container finished" podID="d3a9f32d-0e80-42ce-81ad-d4c08790fccb" containerID="5ae034fee90524865e2694fe0c37da036b6443201ee3379f601cb1928263a4fb" exitCode=0 Nov 03 17:46:02.993926 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.993818 5805 generic.go:356] "Generic (PLEG): container finished" podID="d3a9f32d-0e80-42ce-81ad-d4c08790fccb" containerID="9a3105728ee2a2ba11b1b035a3c07d822ecac7ec085b41978efc015e96be8865" exitCode=0 Nov 03 17:46:02.993926 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.993827 5805 generic.go:356] "Generic (PLEG): container finished" podID="d3a9f32d-0e80-42ce-81ad-d4c08790fccb" containerID="4114ff5cd7a69108d4b88c0578c3be346416f484adef1c25715d7c22558340f8" exitCode=0 Nov 03 17:46:02.993926 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.993833 5805 generic.go:356] "Generic (PLEG): container finished" podID="d3a9f32d-0e80-42ce-81ad-d4c08790fccb" containerID="3ced9032244d3922a3430c576f33b787eb8c9336538a8488e80bd7069b51734b" exitCode=0 Nov 03 17:46:02.993926 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:02.993839 5805 generic.go:356] "Generic (PLEG): container finished" podID="d3a9f32d-0e80-42ce-81ad-d4c08790fccb" containerID="08f89d11852613c79ae48fe38669f896b846334ef8eb23630f28d2fcf6a87094" exitCode=0 Nov 03 17:46:03.000935 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.000910 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 17:46:03.001301 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.001271 5805 generic.go:356] "Generic (PLEG): container finished" podID="6a74bba5-bc9c-4ae1-9754-8887c5bf23cc" containerID="340a940ba8b8964d841361a71a7d138c5ba910906e6863f9cfc28e6dce4b6469" exitCode=1 Nov 03 17:46:03.008561 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.008533 5805 generic.go:356] "Generic (PLEG): container finished" podID="a8abced1-c904-4e84-b62e-96290db6fa8e" containerID="603181325bf1708b995dc9c57b8b797ec0ff81577177bc5c5cfa29133f7bae26" exitCode=0 Nov 03 17:46:03.015240 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.015222 5805 generic.go:356] "Generic (PLEG): container finished" podID="79621dd6debcc143ca37704603624014" containerID="9480f947e0b0ddfc27d46853e2fc647f346bb07725948977c119f697e270f3ac" exitCode=0 Nov 03 17:46:03.020118 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.020062 5805 generic.go:356] "Generic (PLEG): container finished" podID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerID="27a60c8c7c4610b479e0c7e24c5cf4dd2054502a61ed64d7bdb7e8f3751d87a9" exitCode=0 Nov 03 17:46:03.039644 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:03.039613 5805 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-164.us-west-2.compute.internal\" not found" Nov 03 17:46:03.055240 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.055219 5805 manager.go:324] Recovery completed Nov 03 17:46:03.069557 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:03.069537 5805 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 17:46:03.075694 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.075677 5805 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:46:03.079643 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.079623 5805 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:46:03.079720 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.079657 5805 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:46:03.079720 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.079669 5805 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:46:03.082981 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.082963 5805 cpu_manager.go:226] "Starting" policy="none" Nov 03 17:46:03.082981 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.082977 5805 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 17:46:03.082981 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.082977 5805 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:46:03.083183 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.082993 5805 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 17:46:03.083183 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.083119 5805 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 17:46:03.083183 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.083131 5805 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 17:46:03.083183 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.083147 5805 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 17:46:03.083183 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.083152 5805 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 17:46:03.083183 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.083157 5805 policy_none.go:50] "Start" Nov 03 17:46:03.083183 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.083166 5805 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 17:46:03.083183 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.083175 5805 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 17:46:03.083415 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.083310 5805 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 17:46:03.083415 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.083317 5805 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 17:46:03.083415 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.083322 5805 policy_none.go:44] "Start" Nov 03 17:46:03.083951 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.083934 5805 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:46:03.084014 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.083963 5805 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:46:03.084014 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.083977 5805 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:46:03.087921 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.087898 5805 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 17:46:03.091300 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.091288 5805 manager.go:341] "Starting Device Plugin manager" Nov 03 17:46:03.091353 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:03.091312 5805 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 17:46:03.091353 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.091320 5805 server.go:87] "Starting device plugin registration server" Nov 03 17:46:03.091608 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.091598 5805 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 17:46:03.091652 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.091610 5805 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 17:46:03.091723 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.091706 5805 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 17:46:03.091874 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.091839 5805 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 17:46:03.091941 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.091887 5805 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 17:46:03.091941 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.091895 5805 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 17:46:03.092036 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.091928 5805 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T17:46:03.091871901Z","UUID":"9b67db3e-6806-4c39-b07c-2f8e2afd9b8a","Handler":null,"Name":"","Endpoint":""} Nov 03 17:46:03.093592 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.093573 5805 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 17:46:03.093679 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.093602 5805 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 17:46:03.094079 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:03.094055 5805 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 17:46:03.094166 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:03.094100 5805 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-164.us-west-2.compute.internal\" not found" Nov 03 17:46:03.130624 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.130604 5805 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:46:03.130739 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.130675 5805 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:46:03.130739 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.130683 5805 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 17:46:03.130739 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.130682 5805 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:46:03.191826 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.191800 5805 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:46:03.192793 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.192778 5805 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:46:03.192870 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.192807 5805 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:46:03.192870 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.192817 5805 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-164.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 17:46:03.192870 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.192869 5805 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:46:03.222481 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.222456 5805 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:46:03.270194 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.270120 5805 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal"] Nov 03 17:46:03.270829 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.270814 5805 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:46:03.270899 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.270818 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" event={"ID":"79621dd6debcc143ca37704603624014","Type":"ContainerStarted","Data":"5d7f4f365c0af7bd9915fd75a3b5b87182707ba405a75d17dd28b1a8e89cdfbe"} Nov 03 17:46:03.270899 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.270860 5805 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:46:03.270899 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.270889 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" event={"ID":"79621dd6debcc143ca37704603624014","Type":"ContainerDied","Data":"9480f947e0b0ddfc27d46853e2fc647f346bb07725948977c119f697e270f3ac"} Nov 03 17:46:03.271050 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.270912 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" event={"ID":"79621dd6debcc143ca37704603624014","Type":"ContainerStarted","Data":"813fa771298c166a6343b5651245a1ed32536335e2e63b6e357d6f4210b18c70"} Nov 03 17:46:03.271050 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.270933 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal" event={"ID":"5df37dbe21c11f3ff8879a1693476b0b","Type":"ContainerStarted","Data":"d885c791a36ee714ebf6c5a7a2576b75becd09ec2562133570cb236c82269e7e"} Nov 03 17:46:03.271050 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.270952 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal" event={"ID":"5df37dbe21c11f3ff8879a1693476b0b","Type":"ContainerStarted","Data":"4357fa06359bdb104dcf16e83dd04927773998373e48cd06ac23b95f7e7441fb"} Nov 03 17:46:03.295909 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.295889 5805 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:46:03.295965 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.295956 5805 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:46:03.328548 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.328528 5805 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 17:46:03.328637 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:03.328565 5805 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:46:03.329442 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.329426 5805 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 17:46:03.329515 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:03.329470 5805 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:46:03.431888 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.431855 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/79621dd6debcc143ca37704603624014-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal\" (UID: \"79621dd6debcc143ca37704603624014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:46:03.431888 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.431887 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79621dd6debcc143ca37704603624014-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal\" (UID: \"79621dd6debcc143ca37704603624014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:46:03.432101 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.431931 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5df37dbe21c11f3ff8879a1693476b0b-config\") pod \"kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal\" (UID: \"5df37dbe21c11f3ff8879a1693476b0b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:46:03.532664 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.532592 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/79621dd6debcc143ca37704603624014-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal\" (UID: \"79621dd6debcc143ca37704603624014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:46:03.532664 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.532538 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/79621dd6debcc143ca37704603624014-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal\" (UID: \"79621dd6debcc143ca37704603624014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:46:03.532812 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.532665 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79621dd6debcc143ca37704603624014-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal\" (UID: \"79621dd6debcc143ca37704603624014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:46:03.532812 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.532683 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5df37dbe21c11f3ff8879a1693476b0b-config\") pod \"kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal\" (UID: \"5df37dbe21c11f3ff8879a1693476b0b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:46:03.532812 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.532711 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5df37dbe21c11f3ff8879a1693476b0b-config\") pod \"kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal\" (UID: \"5df37dbe21c11f3ff8879a1693476b0b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:46:03.532812 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.532729 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79621dd6debcc143ca37704603624014-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal\" (UID: \"79621dd6debcc143ca37704603624014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:46:03.925876 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:03.925847 5805 apiserver.go:52] "Watching apiserver" Nov 03 17:46:04.023361 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.023331 5805 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 17:46:04.023964 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.023937 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0","openshift-multus/network-metrics-daemon-fs77w","openshift-image-registry/node-ca-xk8dp","openshift-monitoring/metrics-server-687b6669ff-vc8sm","openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc","openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr","openshift-insights/insights-runtime-extractor-rbt65","openshift-monitoring/prometheus-k8s-0","openshift-monitoring/telemeter-client-67d8bb587d-lcgff","kube-system/kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-8p8q8","kube-system/global-pull-secret-syncer-8zpkh","openshift-ingress-canary/ingress-canary-fkhbf","openshift-ovn-kubernetes/ovnkube-node-jjq2h","openshift-dns/dns-default-khc2s","openshift-dns/node-resolver-qjbks","openshift-monitoring/node-exporter-sf77n","kube-system/konnectivity-agent-8b7gg","openshift-cluster-node-tuning-operator/tuned-xm4q9","openshift-multus/multus-4hhwj","openshift-network-operator/iptables-alerter-njt29","openshift-multus/multus-additional-cni-plugins-fwttb"] Nov 03 17:46:04.039531 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.039503 5805 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:46:04.039712 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.039696 5805 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:46:04.076175 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.076152 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6lzh9\"" Nov 03 17:46:04.076175 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.076156 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sq9vz\"" Nov 03 17:46:04.076175 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.076198 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-j4vc9\"" Nov 03 17:46:04.076533 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.076326 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 17:46:04.076533 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.076370 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5hxz7\"" Nov 03 17:46:04.076533 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.076380 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-bmd8j\"" Nov 03 17:46:04.076533 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.076450 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8lnxv\"" Nov 03 17:46:04.076533 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.076502 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 17:46:04.076788 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.076680 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 17:46:04.076840 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.076796 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 17:46:04.076840 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.076801 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mk8dz\"" Nov 03 17:46:04.076840 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.076817 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m6lsw\"" Nov 03 17:46:04.076980 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.076829 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 17:46:04.076980 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.076861 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 17:46:04.076980 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.076956 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 17:46:04.077174 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.077160 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 17:46:04.077247 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.077200 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 17:46:04.077426 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.077413 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 17:46:04.077492 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.077476 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 17:46:04.077577 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.077556 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 17:46:04.077698 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.077585 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 17:46:04.077698 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.077587 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 17:46:04.077698 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.077680 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 17:46:04.077698 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.077692 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 17:46:04.077894 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.077788 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 17:46:04.077894 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.077852 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 17:46:04.077969 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.077925 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 17:46:04.078017 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.077993 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 17:46:04.078017 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.078003 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 17:46:04.078017 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.078010 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 17:46:04.078017 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.077997 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f85t9\"" Nov 03 17:46:04.078181 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.078018 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 17:46:04.078181 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.078036 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8w5gn\"" Nov 03 17:46:04.078181 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.078077 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 17:46:04.078181 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.078081 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 17:46:04.078181 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.078141 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 17:46:04.078181 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.078158 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 17:46:04.078373 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.078283 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 17:46:04.078373 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.078324 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 17:46:04.078373 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.078341 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 17:46:04.078485 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.078412 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z56rs\"" Nov 03 17:46:04.078485 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.078429 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 17:46:04.078557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.078517 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 17:46:04.078639 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.078626 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l7l2n\"" Nov 03 17:46:04.114368 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.114350 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 17:46:04.114479 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.114464 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nvxmw\"" Nov 03 17:46:04.117135 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.117124 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 17:46:04.118003 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.117988 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 17:46:04.118230 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.118214 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 17:46:04.118341 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.118322 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-556wq\"" Nov 03 17:46:04.118341 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.118327 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 17:46:04.118486 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.118388 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pvbgk\"" Nov 03 17:46:04.118581 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.118567 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 17:46:04.118624 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.118573 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 17:46:04.118624 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.118572 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 17:46:04.118682 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.118623 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 17:46:04.125361 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.125341 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 17:46:04.125429 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.125413 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 17:46:04.125520 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.125506 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 17:46:04.125565 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.125551 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 17:46:04.125743 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.125727 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 17:46:04.125828 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.125812 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 17:46:04.125882 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.125830 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 17:46:04.125930 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.125840 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 17:46:04.125930 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.125897 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 17:46:04.126140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.126125 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 17:46:04.126140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.126136 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-251ja618v50s1\"" Nov 03 17:46:04.126248 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.126130 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 17:46:04.126248 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.126229 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bungpse31jqe3\"" Nov 03 17:46:04.126321 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.126314 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-k2vg5\"" Nov 03 17:46:04.126866 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.126852 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 17:46:04.127075 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.127061 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 17:46:04.130331 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.130316 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 17:46:04.130656 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.130643 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 17:46:04.130764 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.130752 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 17:46:04.130872 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.130857 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 17:46:04.131689 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.131671 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 17:46:04.132089 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.131840 5805 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 17:46:04.132089 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.131895 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 17:46:04.132089 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:04.131897 5805 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:46:04.132089 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.132072 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 17:46:04.132089 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.132061 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 17:46:04.132289 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.132159 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 17:46:04.132319 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.132302 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7d5fs\"" Nov 03 17:46:04.134615 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.134600 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 17:46:04.134824 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.134806 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 17:46:04.134886 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.134844 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jtjm7\"" Nov 03 17:46:04.134886 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.134858 5805 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 17:46:04.134969 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.134903 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 17:46:04.134969 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.134910 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 17:46:04.134969 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:04.134902 5805 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-164.us-west-2.compute.internal" Nov 03 17:46:04.134969 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.134922 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 17:46:04.134969 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.134922 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 17:46:04.134969 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.134910 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-q2hxk\"" Nov 03 17:46:04.135181 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.135164 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 17:46:04.136150 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.136127 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2c2r\" (UniqueName: \"kubernetes.io/projected/c5890bb8-3e6a-4250-8ccb-50771cd112c0-kube-api-access-l2c2r\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:46:04.136229 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.136162 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-config\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.136318 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.136299 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.136351 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.136338 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-etc-openvswitch\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.136390 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.136367 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-run-k8s-cni-cncf-io\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.136483 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.136460 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/62507a50-2bff-4f45-82e7-aca45c032546-multus-daemon-config\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.136552 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.136532 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2b3783-58ff-4bb7-b713-6da60023f5ef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.136606 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.136565 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.136606 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.136592 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-kubelet\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.136699 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.136616 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzmxx\" (UniqueName: \"kubernetes.io/projected/815e56fd-a11f-4a53-8d85-bfc073f33ea8-kube-api-access-lzmxx\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:46:04.136699 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.136661 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-run-multus-certs\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.136699 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.136688 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc524cbd-84c3-4255-a208-26fd55af457e-iptables-alerter-script\") pod \"iptables-alerter-njt29\" (UID: \"dc524cbd-84c3-4255-a208-26fd55af457e\") " pod="openshift-network-operator/iptables-alerter-njt29" Nov 03 17:46:04.136844 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.136730 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/62507a50-2bff-4f45-82e7-aca45c032546-multus-daemon-config\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.136844 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.136740 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc524cbd-84c3-4255-a208-26fd55af457e-host-slash\") pod \"iptables-alerter-njt29\" (UID: \"dc524cbd-84c3-4255-a208-26fd55af457e\") " pod="openshift-network-operator/iptables-alerter-njt29" Nov 03 17:46:04.136844 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.136776 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-accelerators-collector-config\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:46:04.136844 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.136811 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.136844 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.136825 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-cni-bin\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.137022 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.136909 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-device-dir\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:46:04.137022 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.136946 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8997w\" (UniqueName: \"kubernetes.io/projected/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-kube-api-access-8997w\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:46:04.137022 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.136994 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-lib-modules\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.137136 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.137036 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d82f08c7-9da6-4969-858f-c8e5b140d27c-secret-metrics-server-tls\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:46:04.137136 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.137052 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc524cbd-84c3-4255-a208-26fd55af457e-iptables-alerter-script\") pod \"iptables-alerter-njt29\" (UID: \"dc524cbd-84c3-4255-a208-26fd55af457e\") " pod="openshift-network-operator/iptables-alerter-njt29" Nov 03 17:46:04.137136 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.137076 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d82f08c7-9da6-4969-858f-c8e5b140d27c-audit-log\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:46:04.137136 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.137093 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmhkc\" (UniqueName: \"kubernetes.io/projected/40fbf434-33db-4779-acf2-a148d4cb8c60-kube-api-access-hmhkc\") pod \"ingress-canary-fkhbf\" (UID: \"40fbf434-33db-4779-acf2-a148d4cb8c60\") " pod="openshift-ingress-canary/ingress-canary-fkhbf" Nov 03 17:46:04.137136 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.137034 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-accelerators-collector-config\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:46:04.137321 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.137131 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-slash\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.137321 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.137265 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-run-openvswitch\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.137321 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.137253 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d82f08c7-9da6-4969-858f-c8e5b140d27c-audit-log\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:46:04.137321 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.137295 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-var-lib-cni-multus\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.137512 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.137358 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/78a8038b-489c-44f5-827f-d4aaebca42f8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-c5q4r\" (UID: \"78a8038b-489c-44f5-827f-d4aaebca42f8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" Nov 03 17:46:04.137512 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.137432 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-config-volume\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.137512 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.137464 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-ovnkube-script-lib\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.137625 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.137497 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-socket-dir\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:46:04.137625 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.137580 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret\") pod \"global-pull-secret-syncer-8zpkh\" (UID: \"c527a13f-da23-4228-a048-4af962c36340\") " pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:46:04.137702 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.137679 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-config-volume\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.137750 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.137735 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-multus-cni-dir\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.137782 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.137765 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-var-lib-cni-bin\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.137883 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.137866 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-ovnkube-script-lib\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.138061 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.138039 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c527a13f-da23-4228-a048-4af962c36340-original-pull-secret\") pod \"global-pull-secret-syncer-8zpkh\" (UID: \"c527a13f-da23-4228-a048-4af962c36340\") " pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:46:04.138237 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.138221 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189-config-volume\") pod \"dns-default-khc2s\" (UID: \"4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189\") " pod="openshift-dns/dns-default-khc2s" Nov 03 17:46:04.138287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.138265 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e2b3783-58ff-4bb7-b713-6da60023f5ef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.138320 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.138292 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c5890bb8-3e6a-4250-8ccb-50771cd112c0-telemeter-client-tls\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:46:04.138355 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.138318 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.138390 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.138354 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.138472 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.138386 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.138472 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.138407 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189-config-volume\") pod \"dns-default-khc2s\" (UID: \"4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189\") " pod="openshift-dns/dns-default-khc2s" Nov 03 17:46:04.138587 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.138539 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e2b3783-58ff-4bb7-b713-6da60023f5ef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.138644 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.138587 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.138644 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.138592 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-multus-conf-dir\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.138644 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.138597 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.138741 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.138641 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9107231-c3e3-4d76-81e6-b276b828c9bc-serviceca\") pod \"node-ca-xk8dp\" (UID: \"c9107231-c3e3-4d76-81e6-b276b828c9bc\") " pod="openshift-image-registry/node-ca-xk8dp" Nov 03 17:46:04.138741 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.138672 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tv8z\" (UniqueName: \"kubernetes.io/projected/c9107231-c3e3-4d76-81e6-b276b828c9bc-kube-api-access-6tv8z\") pod \"node-ca-xk8dp\" (UID: \"c9107231-c3e3-4d76-81e6-b276b828c9bc\") " pod="openshift-image-registry/node-ca-xk8dp" Nov 03 17:46:04.138741 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.138702 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5m9x\" (UniqueName: \"kubernetes.io/projected/d82f08c7-9da6-4969-858f-c8e5b140d27c-kube-api-access-v5m9x\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:46:04.138741 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.138719 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c5890bb8-3e6a-4250-8ccb-50771cd112c0-telemeter-client-tls\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:46:04.138741 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.138729 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.138970 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.138766 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-sysconfig\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.138970 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.138791 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-systemd\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.138970 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.138819 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5890bb8-3e6a-4250-8ccb-50771cd112c0-metrics-client-ca\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:46:04.138970 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.138850 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.138970 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.138874 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-kubernetes\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.138970 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.138913 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83572d76-31ae-42e0-b7d4-acc08df84a9c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:46:04.138970 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.138947 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.139280 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.139030 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.139280 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.139142 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9107231-c3e3-4d76-81e6-b276b828c9bc-serviceca\") pod \"node-ca-xk8dp\" (UID: \"c9107231-c3e3-4d76-81e6-b276b828c9bc\") " pod="openshift-image-registry/node-ca-xk8dp" Nov 03 17:46:04.139280 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.139155 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.139280 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.139193 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-cni-binary-copy\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:46:04.139280 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.139143 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5890bb8-3e6a-4250-8ccb-50771cd112c0-metrics-client-ca\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:46:04.139280 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.139234 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83572d76-31ae-42e0-b7d4-acc08df84a9c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:46:04.139498 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.139344 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.139498 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.139427 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-run-systemd\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.139591 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.139491 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c527a13f-da23-4228-a048-4af962c36340-kubelet-config\") pod \"global-pull-secret-syncer-8zpkh\" (UID: \"c527a13f-da23-4228-a048-4af962c36340\") " pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:46:04.139591 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.139537 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnv9j\" (UniqueName: \"kubernetes.io/projected/83572d76-31ae-42e0-b7d4-acc08df84a9c-kube-api-access-mnv9j\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:46:04.139591 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.139584 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-ovnkube-config\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.139713 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.139630 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.139713 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.139652 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189-tmp-dir\") pod \"dns-default-khc2s\" (UID: \"4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189\") " pod="openshift-dns/dns-default-khc2s" Nov 03 17:46:04.139713 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.139658 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-cni-binary-copy\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:46:04.139807 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.139785 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:46:04.139807 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.139794 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-ovnkube-config\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.139878 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.139838 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-modprobe-d\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.139944 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.139917 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lcbs\" (UniqueName: \"kubernetes.io/projected/32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93-kube-api-access-6lcbs\") pod \"node-resolver-qjbks\" (UID: \"32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93\") " pod="openshift-dns/node-resolver-qjbks" Nov 03 17:46:04.139944 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.139919 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189-tmp-dir\") pod \"dns-default-khc2s\" (UID: \"4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189\") " pod="openshift-dns/dns-default-khc2s" Nov 03 17:46:04.140049 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.139960 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:46:04.140049 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.140038 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18c90190-94ae-4270-b735-a46d3aaa3f42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:46:04.140145 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.140085 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e2b3783-58ff-4bb7-b713-6da60023f5ef-config-out\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.140145 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.140110 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.140233 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.140144 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-env-overrides\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.140233 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.140163 5805 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 17:46:04.140233 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.140169 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljxz9\" (UniqueName: \"kubernetes.io/projected/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189-kube-api-access-ljxz9\") pod \"dns-default-khc2s\" (UID: \"4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189\") " pod="openshift-dns/dns-default-khc2s" Nov 03 17:46:04.140233 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.140200 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e2b3783-58ff-4bb7-b713-6da60023f5ef-config-out\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.140746 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.140722 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18c90190-94ae-4270-b735-a46d3aaa3f42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:46:04.140746 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.140732 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-sysctl-d\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.140899 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.140772 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.140899 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.140785 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5flr\" (UniqueName: \"kubernetes.io/projected/18c90190-94ae-4270-b735-a46d3aaa3f42-kube-api-access-d5flr\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:46:04.140899 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.140827 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-env-overrides\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.140899 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.140842 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8abced1-c904-4e84-b62e-96290db6fa8e-root\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:46:04.141097 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.140973 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-wtmp\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:46:04.141097 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141015 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb74w\" (UniqueName: \"kubernetes.io/projected/a8abced1-c904-4e84-b62e-96290db6fa8e-kube-api-access-tb74w\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:46:04.141202 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141187 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.141273 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141228 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/83572d76-31ae-42e0-b7d4-acc08df84a9c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:46:04.141273 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141265 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8abced1-c904-4e84-b62e-96290db6fa8e-metrics-client-ca\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:46:04.141375 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141291 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-web-config\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.141375 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141327 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5890bb8-3e6a-4250-8ccb-50771cd112c0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:46:04.141375 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141362 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.141539 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141414 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f7aa991-10ec-45ff-ac60-547adf0501a2-tmp\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.141539 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141443 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jmjl\" (UniqueName: \"kubernetes.io/projected/0f7aa991-10ec-45ff-ac60-547adf0501a2-kube-api-access-8jmjl\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.141539 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141461 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.141539 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141468 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.141539 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141473 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40fbf434-33db-4779-acf2-a148d4cb8c60-cert\") pod \"ingress-canary-fkhbf\" (UID: \"40fbf434-33db-4779-acf2-a148d4cb8c60\") " pod="openshift-ingress-canary/ingress-canary-fkhbf" Nov 03 17:46:04.141763 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141573 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5890bb8-3e6a-4250-8ccb-50771cd112c0-serving-certs-ca-bundle\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:46:04.141763 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141610 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-web-config\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.141763 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141644 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-run-ovn-kubernetes\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.141763 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141684 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdzrj\" (UniqueName: \"kubernetes.io/projected/dc524cbd-84c3-4255-a208-26fd55af457e-kube-api-access-kdzrj\") pod \"iptables-alerter-njt29\" (UID: \"dc524cbd-84c3-4255-a208-26fd55af457e\") " pod="openshift-network-operator/iptables-alerter-njt29" Nov 03 17:46:04.141763 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141728 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:46:04.141980 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141766 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18c90190-94ae-4270-b735-a46d3aaa3f42-crio-socket\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:46:04.141980 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141795 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/83572d76-31ae-42e0-b7d4-acc08df84a9c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:46:04.141980 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141826 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c5890bb8-3e6a-4250-8ccb-50771cd112c0-federate-client-tls\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:46:04.141980 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141861 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-cnibin\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.141980 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141889 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8abced1-c904-4e84-b62e-96290db6fa8e-metrics-client-ca\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:46:04.141980 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141904 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48n9p\" (UniqueName: \"kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p\") pod \"network-check-target-8p8q8\" (UID: \"d2c0ef79-ea05-4061-b4dd-31edac551213\") " pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:46:04.141980 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141943 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.142332 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.141993 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/604769b0-503d-444f-983e-265898035e90-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.142332 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142207 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c5890bb8-3e6a-4250-8ccb-50771cd112c0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:46:04.142332 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142271 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.142332 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142308 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-etc-kubernetes\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.142584 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142342 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189-metrics-tls\") pod \"dns-default-khc2s\" (UID: \"4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189\") " pod="openshift-dns/dns-default-khc2s" Nov 03 17:46:04.142584 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142355 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/40fbf434-33db-4779-acf2-a148d4cb8c60-cert\") pod \"ingress-canary-fkhbf\" (UID: \"40fbf434-33db-4779-acf2-a148d4cb8c60\") " pod="openshift-ingress-canary/ingress-canary-fkhbf" Nov 03 17:46:04.142584 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142377 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:46:04.142584 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142422 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-sysctl-conf\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.142584 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142453 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-tls\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:46:04.142584 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142485 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c5890bb8-3e6a-4250-8ccb-50771cd112c0-secret-telemeter-client\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:46:04.142584 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142514 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-ovn-node-metrics-cert\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.142584 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142572 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-etc-selinux\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:46:04.142950 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142598 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-run-netns\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.142950 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142602 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-web-config\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.142950 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142650 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9107231-c3e3-4d76-81e6-b276b828c9bc-host\") pod \"node-ca-xk8dp\" (UID: \"c9107231-c3e3-4d76-81e6-b276b828c9bc\") " pod="openshift-image-registry/node-ca-xk8dp" Nov 03 17:46:04.142950 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142680 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-var-lib-kubelet\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.142950 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142711 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d82f08c7-9da6-4969-858f-c8e5b140d27c-client-ca-bundle\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:46:04.142950 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142740 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.142950 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142769 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa56a595-a9ef-431d-b176-41cb1cf382d7-konnectivity-ca\") pod \"konnectivity-agent-8b7gg\" (UID: \"fa56a595-a9ef-431d-b176-41cb1cf382d7\") " pod="kube-system/konnectivity-agent-8b7gg" Nov 03 17:46:04.142950 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142800 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18c90190-94ae-4270-b735-a46d3aaa3f42-data-volume\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:46:04.142950 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142822 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5890bb8-3e6a-4250-8ccb-50771cd112c0-serving-certs-ca-bundle\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:46:04.142950 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142832 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.142950 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142842 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.142950 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142879 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-run-netns\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.142950 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142942 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f7aa991-10ec-45ff-ac60-547adf0501a2-tmp\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.143587 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.142982 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-var-lib-openvswitch\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.143587 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.143025 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78a8038b-489c-44f5-827f-d4aaebca42f8-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-c5q4r\" (UID: \"78a8038b-489c-44f5-827f-d4aaebca42f8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" Nov 03 17:46:04.143587 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.143078 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5890bb8-3e6a-4250-8ccb-50771cd112c0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:46:04.143587 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.143167 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-tls\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:46:04.143587 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.143318 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.143587 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.143492 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa56a595-a9ef-431d-b176-41cb1cf382d7-konnectivity-ca\") pod \"konnectivity-agent-8b7gg\" (UID: \"fa56a595-a9ef-431d-b176-41cb1cf382d7\") " pod="kube-system/konnectivity-agent-8b7gg" Nov 03 17:46:04.143587 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.143549 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c5890bb8-3e6a-4250-8ccb-50771cd112c0-federate-client-tls\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:46:04.143907 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.143699 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d82f08c7-9da6-4969-858f-c8e5b140d27c-client-ca-bundle\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:46:04.143958 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.143940 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-ovn-node-metrics-cert\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.144086 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144049 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.144159 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144121 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:46:04.144159 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144148 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/83572d76-31ae-42e0-b7d4-acc08df84a9c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:46:04.144267 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144171 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-log-socket\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.144267 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144210 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:46:04.144267 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144244 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-sys\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.144480 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144308 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sm4t\" (UniqueName: \"kubernetes.io/projected/78a8038b-489c-44f5-827f-d4aaebca42f8-kube-api-access-9sm4t\") pod \"openshift-state-metrics-7bd7489957-c5q4r\" (UID: \"78a8038b-489c-44f5-827f-d4aaebca42f8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" Nov 03 17:46:04.144480 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144316 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/83572d76-31ae-42e0-b7d4-acc08df84a9c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:46:04.144480 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144344 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e2b3783-58ff-4bb7-b713-6da60023f5ef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.144480 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144377 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.144480 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144383 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/78a8038b-489c-44f5-827f-d4aaebca42f8-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-c5q4r\" (UID: \"78a8038b-489c-44f5-827f-d4aaebca42f8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" Nov 03 17:46:04.144480 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144432 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-systemd-units\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.144480 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144460 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-registration-dir\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:46:04.144820 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144492 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-cnibin\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:46:04.144820 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144567 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.144820 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144577 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.144820 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144590 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.144820 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144624 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-multus-socket-dir-parent\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.144820 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144659 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83572d76-31ae-42e0-b7d4-acc08df84a9c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:46:04.144820 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144686 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa56a595-a9ef-431d-b176-41cb1cf382d7-agent-certs\") pod \"konnectivity-agent-8b7gg\" (UID: \"fa56a595-a9ef-431d-b176-41cb1cf382d7\") " pod="kube-system/konnectivity-agent-8b7gg" Nov 03 17:46:04.144820 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144719 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d82f08c7-9da6-4969-858f-c8e5b140d27c-metrics-server-audit-profiles\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:46:04.144820 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144749 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-system-cni-dir\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.144820 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144766 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189-metrics-tls\") pod \"dns-default-khc2s\" (UID: \"4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189\") " pod="openshift-dns/dns-default-khc2s" Nov 03 17:46:04.144820 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144782 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/83572d76-31ae-42e0-b7d4-acc08df84a9c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:46:04.144820 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144784 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/604769b0-503d-444f-983e-265898035e90-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.145359 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144827 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18c90190-94ae-4270-b735-a46d3aaa3f42-data-volume\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:46:04.145359 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144894 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:46:04.145359 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.144981 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d82f08c7-9da6-4969-858f-c8e5b140d27c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:46:04.145359 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.145028 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/604769b0-503d-444f-983e-265898035e90-config-out\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.145359 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.145063 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.145359 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.145074 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/83572d76-31ae-42e0-b7d4-acc08df84a9c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:46:04.145359 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.145097 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:46:04.145359 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.145100 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6dcg\" (UniqueName: \"kubernetes.io/projected/62507a50-2bff-4f45-82e7-aca45c032546-kube-api-access-q6dcg\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.145359 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.145176 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/604769b0-503d-444f-983e-265898035e90-config-out\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.145359 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.145181 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-textfile\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:46:04.145359 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.145262 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-textfile\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:46:04.145359 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.145350 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d82f08c7-9da6-4969-858f-c8e5b140d27c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:46:04.145926 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.145367 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e2b3783-58ff-4bb7-b713-6da60023f5ef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.145926 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.145569 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.145926 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.145605 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnmr6\" (UniqueName: \"kubernetes.io/projected/604769b0-503d-444f-983e-265898035e90-kube-api-access-xnmr6\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.145926 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.145624 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.145926 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.145697 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs\") pod \"network-metrics-daemon-fs77w\" (UID: \"0ba1ef12-3f5d-4449-8125-4a1921ec2990\") " pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:46:04.145926 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.145746 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-sys-fs\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:46:04.145926 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.145780 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-var-lib-kubelet\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.145926 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.145818 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18c90190-94ae-4270-b735-a46d3aaa3f42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:46:04.145926 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.145842 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d82f08c7-9da6-4969-858f-c8e5b140d27c-metrics-server-audit-profiles\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:46:04.145926 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.145848 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa56a595-a9ef-431d-b176-41cb1cf382d7-agent-certs\") pod \"konnectivity-agent-8b7gg\" (UID: \"fa56a595-a9ef-431d-b176-41cb1cf382d7\") " pod="kube-system/konnectivity-agent-8b7gg" Nov 03 17:46:04.146448 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.146002 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.146448 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.146040 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gznl6\" (UniqueName: \"kubernetes.io/projected/0ba1ef12-3f5d-4449-8125-4a1921ec2990-kube-api-access-gznl6\") pod \"network-metrics-daemon-fs77w\" (UID: \"0ba1ef12-3f5d-4449-8125-4a1921ec2990\") " pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:46:04.146448 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.146074 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-run\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.146448 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.146104 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw4rq\" (UniqueName: \"kubernetes.io/projected/4e2b3783-58ff-4bb7-b713-6da60023f5ef-kube-api-access-hw4rq\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.146448 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.146136 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d82f08c7-9da6-4969-858f-c8e5b140d27c-secret-metrics-server-client-certs\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:46:04.146448 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.146190 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78a8038b-489c-44f5-827f-d4aaebca42f8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-c5q4r\" (UID: \"78a8038b-489c-44f5-827f-d4aaebca42f8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" Nov 03 17:46:04.146448 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.146225 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-node-log\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.146448 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.146252 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ba1ef12-3f5d-4449-8125-4a1921ec2990-metrics-certs\") pod \"network-metrics-daemon-fs77w\" (UID: \"0ba1ef12-3f5d-4449-8125-4a1921ec2990\") " pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:46:04.146448 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.146367 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2jll\" (UniqueName: \"kubernetes.io/projected/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-kube-api-access-t2jll\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.146857 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.146817 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18c90190-94ae-4270-b735-a46d3aaa3f42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:46:04.146894 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.146829 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-hostroot\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.147105 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.147082 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d82f08c7-9da6-4969-858f-c8e5b140d27c-secret-metrics-server-client-certs\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:46:04.147169 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.147088 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-host\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.147203 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.147181 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93-tmp-dir\") pod \"node-resolver-qjbks\" (UID: \"32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93\") " pod="openshift-dns/node-resolver-qjbks" Nov 03 17:46:04.147269 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.147257 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93-tmp-dir\") pod \"node-resolver-qjbks\" (UID: \"32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93\") " pod="openshift-dns/node-resolver-qjbks" Nov 03 17:46:04.147299 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.147273 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4e2b3783-58ff-4bb7-b713-6da60023f5ef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.147350 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.147334 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8abced1-c904-4e84-b62e-96290db6fa8e-sys\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:46:04.147729 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.147713 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-tuned\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.147799 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.147783 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4e2b3783-58ff-4bb7-b713-6da60023f5ef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.147799 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.147792 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93-hosts-file\") pod \"node-resolver-qjbks\" (UID: \"32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93\") " pod="openshift-dns/node-resolver-qjbks" Nov 03 17:46:04.147904 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.147819 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-tuned\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.147904 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.147853 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c527a13f-da23-4228-a048-4af962c36340-dbus\") pod \"global-pull-secret-syncer-8zpkh\" (UID: \"c527a13f-da23-4228-a048-4af962c36340\") " pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:46:04.147991 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.147915 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62507a50-2bff-4f45-82e7-aca45c032546-cni-binary-copy\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.147991 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.147933 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-run-ovn\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.147991 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.147948 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-cni-netd\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.147991 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.147964 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-system-cni-dir\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:46:04.148155 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.148059 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62507a50-2bff-4f45-82e7-aca45c032546-cni-binary-copy\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.163779 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.163757 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 17:46:04.163779 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.163770 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 17:46:04.163896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.163832 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 17:46:04.164172 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.164156 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 17:46:04.164172 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.164167 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 17:46:04.164598 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.164582 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 17:46:04.164598 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.164596 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 17:46:04.165249 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.165234 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-vlppl\"" Nov 03 17:46:04.165457 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.165444 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.166722 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.166705 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-config\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.167604 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.167587 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/78a8038b-489c-44f5-827f-d4aaebca42f8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-c5q4r\" (UID: \"78a8038b-489c-44f5-827f-d4aaebca42f8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" Nov 03 17:46:04.167684 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.167614 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d82f08c7-9da6-4969-858f-c8e5b140d27c-secret-metrics-server-tls\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:46:04.167684 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.167669 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/78a8038b-489c-44f5-827f-d4aaebca42f8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-c5q4r\" (UID: \"78a8038b-489c-44f5-827f-d4aaebca42f8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" Nov 03 17:46:04.169840 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.169826 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.172190 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.172176 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xfh9g\"" Nov 03 17:46:04.172269 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.172242 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.172785 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.172766 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 17:46:04.172873 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.172800 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-z8nlf\"" Nov 03 17:46:04.173785 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.173768 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5890bb8-3e6a-4250-8ccb-50771cd112c0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:46:04.173844 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.173824 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 17:46:04.173967 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.173952 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 17:46:04.174281 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.174268 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 17:46:04.174381 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.174366 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 17:46:04.174452 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.174431 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 17:46:04.174486 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.174438 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 17:46:04.175513 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.175498 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83572d76-31ae-42e0-b7d4-acc08df84a9c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:46:04.177842 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.177801 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.177842 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.177804 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2b3783-58ff-4bb7-b713-6da60023f5ef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.178887 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.178868 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.183147 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.183122 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-web-config\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.184228 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.184211 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c5890bb8-3e6a-4250-8ccb-50771cd112c0-secret-telemeter-client\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:46:04.193035 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.193009 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzmxx\" (UniqueName: \"kubernetes.io/projected/815e56fd-a11f-4a53-8d85-bfc073f33ea8-kube-api-access-lzmxx\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:46:04.193131 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.193074 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2c2r\" (UniqueName: \"kubernetes.io/projected/c5890bb8-3e6a-4250-8ccb-50771cd112c0-kube-api-access-l2c2r\") pod \"telemeter-client-67d8bb587d-lcgff\" (UID: \"c5890bb8-3e6a-4250-8ccb-50771cd112c0\") " pod="openshift-monitoring/telemeter-client-67d8bb587d-lcgff" Nov 03 17:46:04.193201 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.193185 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmhkc\" (UniqueName: \"kubernetes.io/projected/40fbf434-33db-4779-acf2-a148d4cb8c60-kube-api-access-hmhkc\") pod \"ingress-canary-fkhbf\" (UID: \"40fbf434-33db-4779-acf2-a148d4cb8c60\") " pod="openshift-ingress-canary/ingress-canary-fkhbf" Nov 03 17:46:04.194826 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.194804 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8997w\" (UniqueName: \"kubernetes.io/projected/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-kube-api-access-8997w\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:46:04.195130 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.195110 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tv8z\" (UniqueName: \"kubernetes.io/projected/c9107231-c3e3-4d76-81e6-b276b828c9bc-kube-api-access-6tv8z\") pod \"node-ca-xk8dp\" (UID: \"c9107231-c3e3-4d76-81e6-b276b828c9bc\") " pod="openshift-image-registry/node-ca-xk8dp" Nov 03 17:46:04.195442 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.195427 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5m9x\" (UniqueName: \"kubernetes.io/projected/d82f08c7-9da6-4969-858f-c8e5b140d27c-kube-api-access-v5m9x\") pod \"metrics-server-687b6669ff-vc8sm\" (UID: \"d82f08c7-9da6-4969-858f-c8e5b140d27c\") " pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:46:04.195792 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.195776 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljxz9\" (UniqueName: \"kubernetes.io/projected/4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189-kube-api-access-ljxz9\") pod \"dns-default-khc2s\" (UID: \"4cd0f35f-4fe3-4ab9-b92f-ca8240e8a189\") " pod="openshift-dns/dns-default-khc2s" Nov 03 17:46:04.196112 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.196094 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnv9j\" (UniqueName: \"kubernetes.io/projected/83572d76-31ae-42e0-b7d4-acc08df84a9c-kube-api-access-mnv9j\") pod \"kube-state-metrics-5dc8d7d89b-2z6hc\" (UID: \"83572d76-31ae-42e0-b7d4-acc08df84a9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2z6hc" Nov 03 17:46:04.196962 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.196945 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lcbs\" (UniqueName: \"kubernetes.io/projected/32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93-kube-api-access-6lcbs\") pod \"node-resolver-qjbks\" (UID: \"32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93\") " pod="openshift-dns/node-resolver-qjbks" Nov 03 17:46:04.198897 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.198877 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb74w\" (UniqueName: \"kubernetes.io/projected/a8abced1-c904-4e84-b62e-96290db6fa8e-kube-api-access-tb74w\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:46:04.198987 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.198969 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdzrj\" (UniqueName: \"kubernetes.io/projected/dc524cbd-84c3-4255-a208-26fd55af457e-kube-api-access-kdzrj\") pod \"iptables-alerter-njt29\" (UID: \"dc524cbd-84c3-4255-a208-26fd55af457e\") " pod="openshift-network-operator/iptables-alerter-njt29" Nov 03 17:46:04.199518 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.199499 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48n9p\" (UniqueName: \"kubernetes.io/projected/d2c0ef79-ea05-4061-b4dd-31edac551213-kube-api-access-48n9p\") pod \"network-check-target-8p8q8\" (UID: \"d2c0ef79-ea05-4061-b4dd-31edac551213\") " pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:46:04.200038 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.200020 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jmjl\" (UniqueName: \"kubernetes.io/projected/0f7aa991-10ec-45ff-ac60-547adf0501a2-kube-api-access-8jmjl\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.201595 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.201575 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6dcg\" (UniqueName: \"kubernetes.io/projected/62507a50-2bff-4f45-82e7-aca45c032546-kube-api-access-q6dcg\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.201842 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.201827 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5flr\" (UniqueName: \"kubernetes.io/projected/18c90190-94ae-4270-b735-a46d3aaa3f42-kube-api-access-d5flr\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:46:04.204224 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.204203 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sm4t\" (UniqueName: \"kubernetes.io/projected/78a8038b-489c-44f5-827f-d4aaebca42f8-kube-api-access-9sm4t\") pod \"openshift-state-metrics-7bd7489957-c5q4r\" (UID: \"78a8038b-489c-44f5-827f-d4aaebca42f8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c5q4r" Nov 03 17:46:04.204467 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.204448 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gznl6\" (UniqueName: \"kubernetes.io/projected/0ba1ef12-3f5d-4449-8125-4a1921ec2990-kube-api-access-gznl6\") pod \"network-metrics-daemon-fs77w\" (UID: \"0ba1ef12-3f5d-4449-8125-4a1921ec2990\") " pod="openshift-multus/network-metrics-daemon-fs77w" Nov 03 17:46:04.204603 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.204587 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw4rq\" (UniqueName: \"kubernetes.io/projected/4e2b3783-58ff-4bb7-b713-6da60023f5ef-kube-api-access-hw4rq\") pod \"alertmanager-main-0\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:04.205360 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.205344 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnmr6\" (UniqueName: \"kubernetes.io/projected/604769b0-503d-444f-983e-265898035e90-kube-api-access-xnmr6\") pod \"prometheus-k8s-0\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:04.206212 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.206191 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2jll\" (UniqueName: \"kubernetes.io/projected/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-kube-api-access-t2jll\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.248962 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.248935 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-slash\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.249076 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.248968 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-run-openvswitch\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.249076 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249009 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-var-lib-cni-multus\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.249076 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249047 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-socket-dir\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:46:04.249076 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249064 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-multus-cni-dir\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.249216 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249067 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-run-openvswitch\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.249216 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249086 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-var-lib-cni-bin\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.249216 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249073 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-slash\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.249216 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249130 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-var-lib-cni-bin\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.249216 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249136 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-socket-dir\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:46:04.249216 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249145 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-multus-cni-dir\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.249216 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249177 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-var-lib-cni-multus\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.249216 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249194 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-multus-conf-dir\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.249216 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249216 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-sysconfig\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249230 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-systemd\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249246 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-kubernetes\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249248 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-multus-conf-dir\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249267 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-sysconfig\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249275 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-run-systemd\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249302 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-run-systemd\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249307 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-kubernetes\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249307 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-systemd\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249325 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c527a13f-da23-4228-a048-4af962c36340-kubelet-config\") pod \"global-pull-secret-syncer-8zpkh\" (UID: \"c527a13f-da23-4228-a048-4af962c36340\") " pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249343 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-modprobe-d\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249366 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-sysctl-d\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249415 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8abced1-c904-4e84-b62e-96290db6fa8e-root\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249420 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c527a13f-da23-4228-a048-4af962c36340-kubelet-config\") pod \"global-pull-secret-syncer-8zpkh\" (UID: \"c527a13f-da23-4228-a048-4af962c36340\") " pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249435 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-wtmp\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249461 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-run-ovn-kubernetes\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249472 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-modprobe-d\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249477 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18c90190-94ae-4270-b735-a46d3aaa3f42-crio-socket\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249509 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-sysctl-d\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249522 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-run-ovn-kubernetes\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249524 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8abced1-c904-4e84-b62e-96290db6fa8e-root\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249526 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8abced1-c904-4e84-b62e-96290db6fa8e-node-exporter-wtmp\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249545 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-cnibin\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249562 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18c90190-94ae-4270-b735-a46d3aaa3f42-crio-socket\") pod \"insights-runtime-extractor-rbt65\" (UID: \"18c90190-94ae-4270-b735-a46d3aaa3f42\") " pod="openshift-insights/insights-runtime-extractor-rbt65" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249571 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-cnibin\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.249557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249576 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-etc-kubernetes\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249593 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249607 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-etc-kubernetes\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249615 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-sysctl-conf\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249631 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249648 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-etc-selinux\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249673 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-run-netns\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249696 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-etc-sysctl-conf\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249697 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9107231-c3e3-4d76-81e6-b276b828c9bc-host\") pod \"node-ca-xk8dp\" (UID: \"c9107231-c3e3-4d76-81e6-b276b828c9bc\") " pod="openshift-image-registry/node-ca-xk8dp" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249700 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-etc-selinux\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249723 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-run-netns\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249723 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-var-lib-kubelet\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249747 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9107231-c3e3-4d76-81e6-b276b828c9bc-host\") pod \"node-ca-xk8dp\" (UID: \"c9107231-c3e3-4d76-81e6-b276b828c9bc\") " pod="openshift-image-registry/node-ca-xk8dp" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249752 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249782 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-run-netns\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249801 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-var-lib-kubelet\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249804 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-var-lib-openvswitch\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249825 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-run-netns\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249822 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249836 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-var-lib-openvswitch\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249841 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-log-socket\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249859 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-log-socket\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249873 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249895 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-sys\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249920 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-systemd-units\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249932 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249947 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-registration-dir\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249966 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-systemd-units\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249970 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-sys\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249972 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-cnibin\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.249995 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-multus-socket-dir-parent\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250004 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-registration-dir\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250004 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-cnibin\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250016 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-system-cni-dir\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250047 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-system-cni-dir\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250052 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-sys-fs\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250052 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-multus-socket-dir-parent\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250074 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-var-lib-kubelet\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250090 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-var-lib-kubelet\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250096 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-sys-fs\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250109 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-run\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250135 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-node-log\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250158 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-hostroot\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250179 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-host\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250188 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-run\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250203 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8abced1-c904-4e84-b62e-96290db6fa8e-sys\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250209 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-node-log\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250219 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-hostroot\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250227 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-host\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250226 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93-hosts-file\") pod \"node-resolver-qjbks\" (UID: \"32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93\") " pod="openshift-dns/node-resolver-qjbks" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250252 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8abced1-c904-4e84-b62e-96290db6fa8e-sys\") pod \"node-exporter-sf77n\" (UID: \"a8abced1-c904-4e84-b62e-96290db6fa8e\") " pod="openshift-monitoring/node-exporter-sf77n" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250254 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c527a13f-da23-4228-a048-4af962c36340-dbus\") pod \"global-pull-secret-syncer-8zpkh\" (UID: \"c527a13f-da23-4228-a048-4af962c36340\") " pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250275 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-run-ovn\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250280 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93-hosts-file\") pod \"node-resolver-qjbks\" (UID: \"32cb83db-b6ab-4d7e-ae38-2fb9b06e2c93\") " pod="openshift-dns/node-resolver-qjbks" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250289 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c527a13f-da23-4228-a048-4af962c36340-dbus\") pod \"global-pull-secret-syncer-8zpkh\" (UID: \"c527a13f-da23-4228-a048-4af962c36340\") " pod="kube-system/global-pull-secret-syncer-8zpkh" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250292 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-cni-netd\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250316 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-system-cni-dir\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250324 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-run-ovn\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250345 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-cni-netd\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250344 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-etc-openvswitch\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250361 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3a9f32d-0e80-42ce-81ad-d4c08790fccb-system-cni-dir\") pod \"multus-additional-cni-plugins-fwttb\" (UID: \"d3a9f32d-0e80-42ce-81ad-d4c08790fccb\") " pod="openshift-multus/multus-additional-cni-plugins-fwttb" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250374 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-etc-openvswitch\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250376 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-run-k8s-cni-cncf-io\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250420 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-kubelet\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250435 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-run-multus-certs\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250460 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-run-multus-certs\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250461 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/62507a50-2bff-4f45-82e7-aca45c032546-host-run-k8s-cni-cncf-io\") pod \"multus-4hhwj\" (UID: \"62507a50-2bff-4f45-82e7-aca45c032546\") " pod="openshift-multus/multus-4hhwj" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250473 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc524cbd-84c3-4255-a208-26fd55af457e-host-slash\") pod \"iptables-alerter-njt29\" (UID: \"dc524cbd-84c3-4255-a208-26fd55af457e\") " pod="openshift-network-operator/iptables-alerter-njt29" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250492 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc524cbd-84c3-4255-a208-26fd55af457e-host-slash\") pod \"iptables-alerter-njt29\" (UID: \"dc524cbd-84c3-4255-a208-26fd55af457e\") " pod="openshift-network-operator/iptables-alerter-njt29" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250495 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-kubelet\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250504 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-cni-bin\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250520 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-device-dir\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250536 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-lib-modules\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.250543 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250584 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a74bba5-bc9c-4ae1-9754-8887c5bf23cc-host-cni-bin\") pod \"ovnkube-node-jjq2h\" (UID: \"6a74bba5-bc9c-4ae1-9754-8887c5bf23cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:04.252503 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250593 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/815e56fd-a11f-4a53-8d85-bfc073f33ea8-device-dir\") pod \"aws-ebs-csi-driver-node-s9dzr\" (UID: \"815e56fd-a11f-4a53-8d85-bfc073f33ea8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s9dzr" Nov 03 17:46:04.252503 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.250629 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0f7aa991-10ec-45ff-ac60-547adf0501a2-lib-modules\") pod \"tuned-xm4q9\" (UID: \"0f7aa991-10ec-45ff-ac60-547adf0501a2\") " pod="openshift-cluster-node-tuning-operator/tuned-xm4q9" Nov 03 17:46:04.326703 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.326674 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:04.783683 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:04.783647 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:05.012138 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:05.012106 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:05.334578 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:05.334543 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:05.405146 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:05.405118 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-khc2s" Nov 03 17:46:05.553363 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:05.553328 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:05.622564 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:05.622499 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:05.848875 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:05.848839 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:06.063184 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:06.063154 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:06.846001 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:06.845967 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:07.048810 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:07.048777 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:07.405665 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:07.405631 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:08.017607 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:08.017577 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:08.032528 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:08.032505 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:12.932826 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:12.932734 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:46:14.029614 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:14.029582 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:14.030088 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:14.029815 5805 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 17:46:15.071897 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:15.071872 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jjq2h" Nov 03 17:46:16.429325 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:16.429296 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-687b6669ff-vc8sm" Nov 03 17:46:17.972199 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:17.972163 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:18.185591 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:18.185556 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:18.395042 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:18.395009 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:21.210459 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:21.210417 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:21.422620 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:21.422583 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:21.620986 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:21.620960 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:21.679453 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:21.679427 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8b7gg" Nov 03 17:46:21.863573 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:21.863544 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:22.078083 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:22.078053 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:22.290573 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:22.290541 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:26.039482 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:26.039445 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 17:46:26.039962 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:26.039902 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="alertmanager" containerID="cri-o://b1b9657c8a13ebb7a0205c090d1b46ae65244bf207919d8590ac7bce749abdd3" gracePeriod=120 Nov 03 17:46:26.040025 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:26.039945 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="kube-rbac-proxy-metric" containerID="cri-o://cedf34711d8e93cd535cb861ff9d77df849558d9a2e82aa9802f50f293de0536" gracePeriod=120 Nov 03 17:46:26.040078 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:26.040002 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="kube-rbac-proxy" containerID="cri-o://8a9f470a2e6a404567af040875d8c349376de276d7f0abab1f4d0860556493af" gracePeriod=120 Nov 03 17:46:26.040078 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:26.040039 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="prom-label-proxy" containerID="cri-o://53cb5b2140417db45fdf8b53a2d7689731199c3df0ac1a7758f4357e3c344bfb" gracePeriod=120 Nov 03 17:46:26.040179 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:26.040057 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="config-reloader" containerID="cri-o://28f766fde6c28ff9b996544936025fd9294d28b2bb76271ba7c3c61bee954dab" gracePeriod=120 Nov 03 17:46:26.040179 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:26.039962 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="kube-rbac-proxy-web" containerID="cri-o://ee72547fd301caeabc0595fa2f411ed58434fd7f5f6607186969b2b82a87d0ad" gracePeriod=120 Nov 03 17:46:26.233632 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:26.233596 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:46:27.126803 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.126771 5805 generic.go:356] "Generic (PLEG): container finished" podID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerID="53cb5b2140417db45fdf8b53a2d7689731199c3df0ac1a7758f4357e3c344bfb" exitCode=0 Nov 03 17:46:27.126803 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.126797 5805 generic.go:356] "Generic (PLEG): container finished" podID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerID="cedf34711d8e93cd535cb861ff9d77df849558d9a2e82aa9802f50f293de0536" exitCode=0 Nov 03 17:46:27.126803 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.126805 5805 generic.go:356] "Generic (PLEG): container finished" podID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerID="8a9f470a2e6a404567af040875d8c349376de276d7f0abab1f4d0860556493af" exitCode=0 Nov 03 17:46:27.126803 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.126812 5805 generic.go:356] "Generic (PLEG): container finished" podID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerID="28f766fde6c28ff9b996544936025fd9294d28b2bb76271ba7c3c61bee954dab" exitCode=0 Nov 03 17:46:27.126803 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.126819 5805 generic.go:356] "Generic (PLEG): container finished" podID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerID="b1b9657c8a13ebb7a0205c090d1b46ae65244bf207919d8590ac7bce749abdd3" exitCode=0 Nov 03 17:46:27.127315 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.126853 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e2b3783-58ff-4bb7-b713-6da60023f5ef","Type":"ContainerDied","Data":"53cb5b2140417db45fdf8b53a2d7689731199c3df0ac1a7758f4357e3c344bfb"} Nov 03 17:46:27.127315 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.126923 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e2b3783-58ff-4bb7-b713-6da60023f5ef","Type":"ContainerDied","Data":"cedf34711d8e93cd535cb861ff9d77df849558d9a2e82aa9802f50f293de0536"} Nov 03 17:46:27.127315 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.126949 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e2b3783-58ff-4bb7-b713-6da60023f5ef","Type":"ContainerDied","Data":"8a9f470a2e6a404567af040875d8c349376de276d7f0abab1f4d0860556493af"} Nov 03 17:46:27.127315 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.126973 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e2b3783-58ff-4bb7-b713-6da60023f5ef","Type":"ContainerDied","Data":"28f766fde6c28ff9b996544936025fd9294d28b2bb76271ba7c3c61bee954dab"} Nov 03 17:46:27.127315 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.126996 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e2b3783-58ff-4bb7-b713-6da60023f5ef","Type":"ContainerDied","Data":"b1b9657c8a13ebb7a0205c090d1b46ae65244bf207919d8590ac7bce749abdd3"} Nov 03 17:46:27.291170 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.291145 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:27.464197 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.464117 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e2b3783-58ff-4bb7-b713-6da60023f5ef-kube-api-access-hw4rq\" (UniqueName: \"kubernetes.io/projected/4e2b3783-58ff-4bb7-b713-6da60023f5ef-kube-api-access-hw4rq\") pod \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " Nov 03 17:46:27.464197 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.464151 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-web-config\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-web-config\") pod \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " Nov 03 17:46:27.464197 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.464178 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e2b3783-58ff-4bb7-b713-6da60023f5ef-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2b3783-58ff-4bb7-b713-6da60023f5ef-alertmanager-trusted-ca-bundle\") pod \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " Nov 03 17:46:27.464197 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.464202 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e2b3783-58ff-4bb7-b713-6da60023f5ef-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e2b3783-58ff-4bb7-b713-6da60023f5ef-metrics-client-ca\") pod \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " Nov 03 17:46:27.464542 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.464222 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-cluster-tls-config\") pod \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " Nov 03 17:46:27.464542 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.464253 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-config-volume\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-config-volume\") pod \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " Nov 03 17:46:27.464542 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.464271 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e2b3783-58ff-4bb7-b713-6da60023f5ef-tls-assets\" (UniqueName: \"kubernetes.io/projected/4e2b3783-58ff-4bb7-b713-6da60023f5ef-tls-assets\") pod \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " Nov 03 17:46:27.464542 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.464319 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy-web\") pod \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " Nov 03 17:46:27.464542 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.464354 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e2b3783-58ff-4bb7-b713-6da60023f5ef-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4e2b3783-58ff-4bb7-b713-6da60023f5ef-alertmanager-main-db\") pod \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " Nov 03 17:46:27.464542 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.464385 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e2b3783-58ff-4bb7-b713-6da60023f5ef-config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e2b3783-58ff-4bb7-b713-6da60023f5ef-config-out\") pod \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " Nov 03 17:46:27.464542 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.464438 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " Nov 03 17:46:27.464542 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.464489 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-main-tls\") pod \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " Nov 03 17:46:27.464542 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.464532 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy\") pod \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\" (UID: \"4e2b3783-58ff-4bb7-b713-6da60023f5ef\") " Nov 03 17:46:27.464997 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.464615 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2b3783-58ff-4bb7-b713-6da60023f5ef-alertmanager-trusted-ca-bundle" pod "4e2b3783-58ff-4bb7-b713-6da60023f5ef" (UID: "4e2b3783-58ff-4bb7-b713-6da60023f5ef"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:46:27.464997 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.464761 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2b3783-58ff-4bb7-b713-6da60023f5ef-metrics-client-ca" pod "4e2b3783-58ff-4bb7-b713-6da60023f5ef" (UID: "4e2b3783-58ff-4bb7-b713-6da60023f5ef"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:46:27.464997 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.464789 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2b3783-58ff-4bb7-b713-6da60023f5ef-alertmanager-main-db" pod "4e2b3783-58ff-4bb7-b713-6da60023f5ef" (UID: "4e2b3783-58ff-4bb7-b713-6da60023f5ef"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:27.464997 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.464806 5805 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2b3783-58ff-4bb7-b713-6da60023f5ef-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:27.467847 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.467820 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2b3783-58ff-4bb7-b713-6da60023f5ef-kube-api-access-hw4rq" pod "4e2b3783-58ff-4bb7-b713-6da60023f5ef" (UID: "4e2b3783-58ff-4bb7-b713-6da60023f5ef"). InnerVolumeSpecName "kube-api-access-hw4rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:46:27.468219 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.468178 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2b3783-58ff-4bb7-b713-6da60023f5ef-tls-assets" pod "4e2b3783-58ff-4bb7-b713-6da60023f5ef" (UID: "4e2b3783-58ff-4bb7-b713-6da60023f5ef"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:46:27.468318 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.468215 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy-web" pod "4e2b3783-58ff-4bb7-b713-6da60023f5ef" (UID: "4e2b3783-58ff-4bb7-b713-6da60023f5ef"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:46:27.468382 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.468114 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e2b3783-58ff-4bb7-b713-6da60023f5ef-config-out" pod "4e2b3783-58ff-4bb7-b713-6da60023f5ef" (UID: "4e2b3783-58ff-4bb7-b713-6da60023f5ef"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:27.468541 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.468521 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-config-volume" pod "4e2b3783-58ff-4bb7-b713-6da60023f5ef" (UID: "4e2b3783-58ff-4bb7-b713-6da60023f5ef"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:46:27.468704 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.468683 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-main-tls" pod "4e2b3783-58ff-4bb7-b713-6da60023f5ef" (UID: "4e2b3783-58ff-4bb7-b713-6da60023f5ef"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:46:27.468778 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.468754 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy" pod "4e2b3783-58ff-4bb7-b713-6da60023f5ef" (UID: "4e2b3783-58ff-4bb7-b713-6da60023f5ef"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:46:27.468873 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.468856 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy-metric" pod "4e2b3783-58ff-4bb7-b713-6da60023f5ef" (UID: "4e2b3783-58ff-4bb7-b713-6da60023f5ef"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:46:27.471907 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.471879 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-cluster-tls-config" pod "4e2b3783-58ff-4bb7-b713-6da60023f5ef" (UID: "4e2b3783-58ff-4bb7-b713-6da60023f5ef"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:46:27.478683 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.478654 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-web-config" pod "4e2b3783-58ff-4bb7-b713-6da60023f5ef" (UID: "4e2b3783-58ff-4bb7-b713-6da60023f5ef"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:46:27.565203 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.565177 5805 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-main-tls\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:27.565203 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.565202 5805 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:27.565203 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.565212 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw4rq\" (UniqueName: \"kubernetes.io/projected/4e2b3783-58ff-4bb7-b713-6da60023f5ef-kube-api-access-hw4rq\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:27.565426 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.565222 5805 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-web-config\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:27.565426 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.565231 5805 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e2b3783-58ff-4bb7-b713-6da60023f5ef-metrics-client-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:27.565426 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.565239 5805 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-cluster-tls-config\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:27.565426 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.565248 5805 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-config-volume\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:27.565426 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.565258 5805 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e2b3783-58ff-4bb7-b713-6da60023f5ef-tls-assets\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:27.565426 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.565267 5805 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:27.565426 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.565275 5805 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4e2b3783-58ff-4bb7-b713-6da60023f5ef-alertmanager-main-db\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:27.565426 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.565285 5805 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e2b3783-58ff-4bb7-b713-6da60023f5ef-config-out\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:27.565426 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:27.565293 5805 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4e2b3783-58ff-4bb7-b713-6da60023f5ef-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:28.134043 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.134006 5805 generic.go:356] "Generic (PLEG): container finished" podID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerID="ee72547fd301caeabc0595fa2f411ed58434fd7f5f6607186969b2b82a87d0ad" exitCode=0 Nov 03 17:46:28.134506 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.134054 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e2b3783-58ff-4bb7-b713-6da60023f5ef","Type":"ContainerDied","Data":"ee72547fd301caeabc0595fa2f411ed58434fd7f5f6607186969b2b82a87d0ad"} Nov 03 17:46:28.134506 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.134081 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.134506 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.134107 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4e2b3783-58ff-4bb7-b713-6da60023f5ef","Type":"ContainerDied","Data":"8417337ed283c083b428775224f475c51445e5bbc02329e40d863a669796181a"} Nov 03 17:46:28.134506 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.134140 5805 scope.go:122] "RemoveContainer" containerID="53cb5b2140417db45fdf8b53a2d7689731199c3df0ac1a7758f4357e3c344bfb" Nov 03 17:46:28.145582 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.145566 5805 scope.go:122] "RemoveContainer" containerID="cedf34711d8e93cd535cb861ff9d77df849558d9a2e82aa9802f50f293de0536" Nov 03 17:46:28.152177 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.152160 5805 scope.go:122] "RemoveContainer" containerID="8a9f470a2e6a404567af040875d8c349376de276d7f0abab1f4d0860556493af" Nov 03 17:46:28.159062 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.158903 5805 scope.go:122] "RemoveContainer" containerID="ee72547fd301caeabc0595fa2f411ed58434fd7f5f6607186969b2b82a87d0ad" Nov 03 17:46:28.165120 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.165098 5805 scope.go:122] "RemoveContainer" containerID="28f766fde6c28ff9b996544936025fd9294d28b2bb76271ba7c3c61bee954dab" Nov 03 17:46:28.171468 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.171451 5805 scope.go:122] "RemoveContainer" containerID="b1b9657c8a13ebb7a0205c090d1b46ae65244bf207919d8590ac7bce749abdd3" Nov 03 17:46:28.177699 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.177683 5805 scope.go:122] "RemoveContainer" containerID="27a60c8c7c4610b479e0c7e24c5cf4dd2054502a61ed64d7bdb7e8f3751d87a9" Nov 03 17:46:28.183933 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.183917 5805 scope.go:122] "RemoveContainer" containerID="53cb5b2140417db45fdf8b53a2d7689731199c3df0ac1a7758f4357e3c344bfb" Nov 03 17:46:28.184164 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:28.184147 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53cb5b2140417db45fdf8b53a2d7689731199c3df0ac1a7758f4357e3c344bfb\": container with ID starting with 53cb5b2140417db45fdf8b53a2d7689731199c3df0ac1a7758f4357e3c344bfb not found: ID does not exist" containerID="53cb5b2140417db45fdf8b53a2d7689731199c3df0ac1a7758f4357e3c344bfb" Nov 03 17:46:28.184215 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.184174 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53cb5b2140417db45fdf8b53a2d7689731199c3df0ac1a7758f4357e3c344bfb"} err="failed to get container status \"53cb5b2140417db45fdf8b53a2d7689731199c3df0ac1a7758f4357e3c344bfb\": rpc error: code = NotFound desc = could not find container \"53cb5b2140417db45fdf8b53a2d7689731199c3df0ac1a7758f4357e3c344bfb\": container with ID starting with 53cb5b2140417db45fdf8b53a2d7689731199c3df0ac1a7758f4357e3c344bfb not found: ID does not exist" Nov 03 17:46:28.184215 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.184204 5805 scope.go:122] "RemoveContainer" containerID="cedf34711d8e93cd535cb861ff9d77df849558d9a2e82aa9802f50f293de0536" Nov 03 17:46:28.184429 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:28.184413 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cedf34711d8e93cd535cb861ff9d77df849558d9a2e82aa9802f50f293de0536\": container with ID starting with cedf34711d8e93cd535cb861ff9d77df849558d9a2e82aa9802f50f293de0536 not found: ID does not exist" containerID="cedf34711d8e93cd535cb861ff9d77df849558d9a2e82aa9802f50f293de0536" Nov 03 17:46:28.184474 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.184433 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cedf34711d8e93cd535cb861ff9d77df849558d9a2e82aa9802f50f293de0536"} err="failed to get container status \"cedf34711d8e93cd535cb861ff9d77df849558d9a2e82aa9802f50f293de0536\": rpc error: code = NotFound desc = could not find container \"cedf34711d8e93cd535cb861ff9d77df849558d9a2e82aa9802f50f293de0536\": container with ID starting with cedf34711d8e93cd535cb861ff9d77df849558d9a2e82aa9802f50f293de0536 not found: ID does not exist" Nov 03 17:46:28.184474 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.184446 5805 scope.go:122] "RemoveContainer" containerID="8a9f470a2e6a404567af040875d8c349376de276d7f0abab1f4d0860556493af" Nov 03 17:46:28.184697 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:28.184679 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a9f470a2e6a404567af040875d8c349376de276d7f0abab1f4d0860556493af\": container with ID starting with 8a9f470a2e6a404567af040875d8c349376de276d7f0abab1f4d0860556493af not found: ID does not exist" containerID="8a9f470a2e6a404567af040875d8c349376de276d7f0abab1f4d0860556493af" Nov 03 17:46:28.184742 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.184703 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a9f470a2e6a404567af040875d8c349376de276d7f0abab1f4d0860556493af"} err="failed to get container status \"8a9f470a2e6a404567af040875d8c349376de276d7f0abab1f4d0860556493af\": rpc error: code = NotFound desc = could not find container \"8a9f470a2e6a404567af040875d8c349376de276d7f0abab1f4d0860556493af\": container with ID starting with 8a9f470a2e6a404567af040875d8c349376de276d7f0abab1f4d0860556493af not found: ID does not exist" Nov 03 17:46:28.184742 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.184719 5805 scope.go:122] "RemoveContainer" containerID="ee72547fd301caeabc0595fa2f411ed58434fd7f5f6607186969b2b82a87d0ad" Nov 03 17:46:28.184923 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:28.184908 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee72547fd301caeabc0595fa2f411ed58434fd7f5f6607186969b2b82a87d0ad\": container with ID starting with ee72547fd301caeabc0595fa2f411ed58434fd7f5f6607186969b2b82a87d0ad not found: ID does not exist" containerID="ee72547fd301caeabc0595fa2f411ed58434fd7f5f6607186969b2b82a87d0ad" Nov 03 17:46:28.184960 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.184928 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee72547fd301caeabc0595fa2f411ed58434fd7f5f6607186969b2b82a87d0ad"} err="failed to get container status \"ee72547fd301caeabc0595fa2f411ed58434fd7f5f6607186969b2b82a87d0ad\": rpc error: code = NotFound desc = could not find container \"ee72547fd301caeabc0595fa2f411ed58434fd7f5f6607186969b2b82a87d0ad\": container with ID starting with ee72547fd301caeabc0595fa2f411ed58434fd7f5f6607186969b2b82a87d0ad not found: ID does not exist" Nov 03 17:46:28.184960 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.184939 5805 scope.go:122] "RemoveContainer" containerID="28f766fde6c28ff9b996544936025fd9294d28b2bb76271ba7c3c61bee954dab" Nov 03 17:46:28.185123 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:28.185106 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f766fde6c28ff9b996544936025fd9294d28b2bb76271ba7c3c61bee954dab\": container with ID starting with 28f766fde6c28ff9b996544936025fd9294d28b2bb76271ba7c3c61bee954dab not found: ID does not exist" containerID="28f766fde6c28ff9b996544936025fd9294d28b2bb76271ba7c3c61bee954dab" Nov 03 17:46:28.185160 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.185127 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f766fde6c28ff9b996544936025fd9294d28b2bb76271ba7c3c61bee954dab"} err="failed to get container status \"28f766fde6c28ff9b996544936025fd9294d28b2bb76271ba7c3c61bee954dab\": rpc error: code = NotFound desc = could not find container \"28f766fde6c28ff9b996544936025fd9294d28b2bb76271ba7c3c61bee954dab\": container with ID starting with 28f766fde6c28ff9b996544936025fd9294d28b2bb76271ba7c3c61bee954dab not found: ID does not exist" Nov 03 17:46:28.185160 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.185138 5805 scope.go:122] "RemoveContainer" containerID="b1b9657c8a13ebb7a0205c090d1b46ae65244bf207919d8590ac7bce749abdd3" Nov 03 17:46:28.185341 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:28.185318 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1b9657c8a13ebb7a0205c090d1b46ae65244bf207919d8590ac7bce749abdd3\": container with ID starting with b1b9657c8a13ebb7a0205c090d1b46ae65244bf207919d8590ac7bce749abdd3 not found: ID does not exist" containerID="b1b9657c8a13ebb7a0205c090d1b46ae65244bf207919d8590ac7bce749abdd3" Nov 03 17:46:28.185341 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.185335 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b9657c8a13ebb7a0205c090d1b46ae65244bf207919d8590ac7bce749abdd3"} err="failed to get container status \"b1b9657c8a13ebb7a0205c090d1b46ae65244bf207919d8590ac7bce749abdd3\": rpc error: code = NotFound desc = could not find container \"b1b9657c8a13ebb7a0205c090d1b46ae65244bf207919d8590ac7bce749abdd3\": container with ID starting with b1b9657c8a13ebb7a0205c090d1b46ae65244bf207919d8590ac7bce749abdd3 not found: ID does not exist" Nov 03 17:46:28.185450 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.185346 5805 scope.go:122] "RemoveContainer" containerID="27a60c8c7c4610b479e0c7e24c5cf4dd2054502a61ed64d7bdb7e8f3751d87a9" Nov 03 17:46:28.185597 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:28.185582 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27a60c8c7c4610b479e0c7e24c5cf4dd2054502a61ed64d7bdb7e8f3751d87a9\": container with ID starting with 27a60c8c7c4610b479e0c7e24c5cf4dd2054502a61ed64d7bdb7e8f3751d87a9 not found: ID does not exist" containerID="27a60c8c7c4610b479e0c7e24c5cf4dd2054502a61ed64d7bdb7e8f3751d87a9" Nov 03 17:46:28.185644 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.185601 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27a60c8c7c4610b479e0c7e24c5cf4dd2054502a61ed64d7bdb7e8f3751d87a9"} err="failed to get container status \"27a60c8c7c4610b479e0c7e24c5cf4dd2054502a61ed64d7bdb7e8f3751d87a9\": rpc error: code = NotFound desc = could not find container \"27a60c8c7c4610b479e0c7e24c5cf4dd2054502a61ed64d7bdb7e8f3751d87a9\": container with ID starting with 27a60c8c7c4610b479e0c7e24c5cf4dd2054502a61ed64d7bdb7e8f3751d87a9 not found: ID does not exist" Nov 03 17:46:28.335633 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.335598 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 17:46:28.381868 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.381760 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 17:46:28.382222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.382203 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 17:46:28.382514 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.382499 5805 cpu_manager.go:414] "containerMap: removing container" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="init-config-reloader" Nov 03 17:46:28.382599 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.382516 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="init-config-reloader" Nov 03 17:46:28.382599 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.382534 5805 cpu_manager.go:414] "containerMap: removing container" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="kube-rbac-proxy" Nov 03 17:46:28.382599 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.382542 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="kube-rbac-proxy" Nov 03 17:46:28.382599 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.382551 5805 cpu_manager.go:414] "containerMap: removing container" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="config-reloader" Nov 03 17:46:28.382599 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.382558 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="config-reloader" Nov 03 17:46:28.382599 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.382573 5805 cpu_manager.go:414] "containerMap: removing container" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="prom-label-proxy" Nov 03 17:46:28.382599 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.382580 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="prom-label-proxy" Nov 03 17:46:28.382599 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.382596 5805 cpu_manager.go:414] "containerMap: removing container" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="kube-rbac-proxy-metric" Nov 03 17:46:28.382975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.382603 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="kube-rbac-proxy-metric" Nov 03 17:46:28.382975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.382624 5805 cpu_manager.go:414] "containerMap: removing container" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="alertmanager" Nov 03 17:46:28.382975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.382632 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="alertmanager" Nov 03 17:46:28.382975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.382652 5805 cpu_manager.go:414] "containerMap: removing container" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="kube-rbac-proxy-web" Nov 03 17:46:28.382975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.382659 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="kube-rbac-proxy-web" Nov 03 17:46:28.382975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.382726 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="kube-rbac-proxy-web" Nov 03 17:46:28.382975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.382739 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="init-config-reloader" Nov 03 17:46:28.382975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.382748 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="kube-rbac-proxy" Nov 03 17:46:28.382975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.382756 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="config-reloader" Nov 03 17:46:28.382975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.382765 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="prom-label-proxy" Nov 03 17:46:28.382975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.382773 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="kube-rbac-proxy-metric" Nov 03 17:46:28.382975 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.382785 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" containerName="alertmanager" Nov 03 17:46:28.388340 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.388281 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.438101 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.438075 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 17:46:28.438229 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.438149 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 17:46:28.438229 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.438167 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 17:46:28.438639 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.438622 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 17:46:28.438781 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.438767 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 17:46:28.438781 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.438772 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 17:46:28.438884 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.438812 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 17:46:28.438884 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.438824 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jtjm7\"" Nov 03 17:46:28.438959 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.438947 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 17:46:28.484378 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.484349 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 17:46:28.573794 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.573761 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.573962 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.573805 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.573962 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.573832 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.574066 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.573952 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.574066 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.573987 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-config-volume\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.574066 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.574009 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.574066 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.574058 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.574219 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.574155 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-web-config\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.574219 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.574187 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.574219 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.574204 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.574328 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.574226 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-config-out\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.574449 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.574335 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.574449 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.574362 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8wxk\" (UniqueName: \"kubernetes.io/projected/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-kube-api-access-b8wxk\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.616494 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.616472 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 17:46:28.675080 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.675009 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-web-config\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.675080 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.675045 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.675080 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.675079 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.675334 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.675108 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-config-out\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.675334 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.675158 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.675334 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.675193 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8wxk\" (UniqueName: \"kubernetes.io/projected/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-kube-api-access-b8wxk\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.675334 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.675228 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.675334 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.675259 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.675334 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.675282 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.675334 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.675314 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.675700 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.675349 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-config-volume\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.675700 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.675381 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.675700 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.675456 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.675700 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.675504 5805 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 17:46:28.677044 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.675917 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.677044 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.676139 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.677044 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.676859 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.678240 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.678216 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-config-out\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.678872 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.678389 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.678872 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.678591 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.678872 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.678718 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-config-volume\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.678872 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.678787 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.678872 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.678830 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.679204 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.679096 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.679778 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.679761 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-web-config\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.680251 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.680231 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.732636 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.732607 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8wxk\" (UniqueName: \"kubernetes.io/projected/c186bca9-0d99-4c9f-a1d9-c8316bd4155f-kube-api-access-b8wxk\") pod \"alertmanager-main-0\" (UID: \"c186bca9-0d99-4c9f-a1d9-c8316bd4155f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:28.973696 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:28.973620 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e2b3783-58ff-4bb7-b713-6da60023f5ef" path="/var/lib/kubelet/pods/4e2b3783-58ff-4bb7-b713-6da60023f5ef/volumes" Nov 03 17:46:29.007073 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:29.007048 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 17:46:29.485632 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:29.485603 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 17:46:29.493071 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:29.493034 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc186bca9_0d99_4c9f_a1d9_c8316bd4155f.slice/crio-5bc0d7dcbcab21844b58f4fb889270e605323fc27bf1b91718471c8d7b1e6b17 WatchSource:0}: Error finding container 5bc0d7dcbcab21844b58f4fb889270e605323fc27bf1b91718471c8d7b1e6b17: Status 404 returned error can't find the container with id 5bc0d7dcbcab21844b58f4fb889270e605323fc27bf1b91718471c8d7b1e6b17 Nov 03 17:46:30.144188 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:30.144157 5805 generic.go:356] "Generic (PLEG): container finished" podID="c186bca9-0d99-4c9f-a1d9-c8316bd4155f" containerID="b250db5d6691b16ff7737371aa8e1653556cfd2449647a75751d0caa36fe6b2b" exitCode=0 Nov 03 17:46:30.144316 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:30.144242 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c186bca9-0d99-4c9f-a1d9-c8316bd4155f","Type":"ContainerDied","Data":"b250db5d6691b16ff7737371aa8e1653556cfd2449647a75751d0caa36fe6b2b"} Nov 03 17:46:30.144316 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:30.144304 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c186bca9-0d99-4c9f-a1d9-c8316bd4155f","Type":"ContainerStarted","Data":"5bc0d7dcbcab21844b58f4fb889270e605323fc27bf1b91718471c8d7b1e6b17"} Nov 03 17:46:31.154146 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:31.154114 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c186bca9-0d99-4c9f-a1d9-c8316bd4155f","Type":"ContainerStarted","Data":"2d9b57b5dc25b896136895379a71e10629242f99976751b1362c613b1e5a58d6"} Nov 03 17:46:31.154628 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:31.154226 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c186bca9-0d99-4c9f-a1d9-c8316bd4155f","Type":"ContainerStarted","Data":"ca6dcec40edfa168c5f0300a753e3dd1ac74c596470d6f6809a2893831a86db0"} Nov 03 17:46:31.154628 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:31.154273 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c186bca9-0d99-4c9f-a1d9-c8316bd4155f","Type":"ContainerStarted","Data":"f644f39f054f218c9f157ddce02f972cf270998e7c1360a303f0e27b7b55973a"} Nov 03 17:46:31.154628 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:31.154333 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c186bca9-0d99-4c9f-a1d9-c8316bd4155f","Type":"ContainerStarted","Data":"d546bb757e37a9d457fa1ebee1d9c38e7bb707b79c78ebdc59dd17657e9b52d3"} Nov 03 17:46:31.154628 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:31.154376 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c186bca9-0d99-4c9f-a1d9-c8316bd4155f","Type":"ContainerStarted","Data":"250626884bedb9bc4a2c5de7600ecbc519d431d4515bbf1cc268975b1f835693"} Nov 03 17:46:31.154628 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:31.154441 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c186bca9-0d99-4c9f-a1d9-c8316bd4155f","Type":"ContainerStarted","Data":"077c53744f607066a841436a89693751a938f9019cddcc34cab961e29e66c500"} Nov 03 17:46:31.277667 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:31.277611 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.266416445 podStartE2EDuration="3.277598029s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.181584ms" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:31.277189917 +0000 UTC m=+28.410818940" watchObservedRunningTime="2026-11-03 17:46:31.277598029 +0000 UTC m=+28.411227035" Nov 03 17:46:31.777098 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:31.777059 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 17:46:31.777528 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:31.777471 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="604769b0-503d-444f-983e-265898035e90" containerName="prometheus" containerID="cri-o://5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2" gracePeriod=600 Nov 03 17:46:31.777742 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:31.777520 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="604769b0-503d-444f-983e-265898035e90" containerName="thanos-sidecar" containerID="cri-o://54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f" gracePeriod=600 Nov 03 17:46:31.777742 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:31.777496 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="604769b0-503d-444f-983e-265898035e90" containerName="kube-rbac-proxy" containerID="cri-o://83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341" gracePeriod=600 Nov 03 17:46:31.777742 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:31.777671 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="604769b0-503d-444f-983e-265898035e90" containerName="kube-rbac-proxy-thanos" containerID="cri-o://85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac" gracePeriod=600 Nov 03 17:46:31.777742 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:31.777616 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="604769b0-503d-444f-983e-265898035e90" containerName="config-reloader" containerID="cri-o://78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562" gracePeriod=600 Nov 03 17:46:31.777948 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:31.777823 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="604769b0-503d-444f-983e-265898035e90" containerName="kube-rbac-proxy-web" containerID="cri-o://ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488" gracePeriod=600 Nov 03 17:46:32.056261 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.056235 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.104991 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.104959 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-0\") pod \"604769b0-503d-444f-983e-265898035e90\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " Nov 03 17:46:32.105117 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.105000 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-kube-rbac-proxy\") pod \"604769b0-503d-444f-983e-265898035e90\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " Nov 03 17:46:32.105117 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.105037 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-db\") pod \"604769b0-503d-444f-983e-265898035e90\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " Nov 03 17:46:32.105117 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.105072 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-tls\") pod \"604769b0-503d-444f-983e-265898035e90\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " Nov 03 17:46:32.105117 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.105101 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-config\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-config\") pod \"604769b0-503d-444f-983e-265898035e90\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " Nov 03 17:46:32.105327 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.105134 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604769b0-503d-444f-983e-265898035e90-kube-api-access-xnmr6\" (UniqueName: \"kubernetes.io/projected/604769b0-503d-444f-983e-265898035e90-kube-api-access-xnmr6\") pod \"604769b0-503d-444f-983e-265898035e90\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " Nov 03 17:46:32.105327 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.105171 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-1\") pod \"604769b0-503d-444f-983e-265898035e90\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " Nov 03 17:46:32.105327 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.105201 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604769b0-503d-444f-983e-265898035e90-tls-assets\" (UniqueName: \"kubernetes.io/projected/604769b0-503d-444f-983e-265898035e90-tls-assets\") pod \"604769b0-503d-444f-983e-265898035e90\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " Nov 03 17:46:32.105327 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.105233 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-grpc-tls\") pod \"604769b0-503d-444f-983e-265898035e90\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " Nov 03 17:46:32.105327 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.105261 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-web-config\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-web-config\") pod \"604769b0-503d-444f-983e-265898035e90\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " Nov 03 17:46:32.105327 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.105295 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-metrics-client-certs\") pod \"604769b0-503d-444f-983e-265898035e90\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " Nov 03 17:46:32.105658 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.105339 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-metrics-client-ca\") pod \"604769b0-503d-444f-983e-265898035e90\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " Nov 03 17:46:32.105658 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.105375 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-serving-certs-ca-bundle\") pod \"604769b0-503d-444f-983e-265898035e90\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " Nov 03 17:46:32.105658 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.105430 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/604769b0-503d-444f-983e-265898035e90-config-out\" (UniqueName: \"kubernetes.io/empty-dir/604769b0-503d-444f-983e-265898035e90-config-out\") pod \"604769b0-503d-444f-983e-265898035e90\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " Nov 03 17:46:32.105658 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.105464 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"604769b0-503d-444f-983e-265898035e90\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " Nov 03 17:46:32.105658 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.105502 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-2\") pod \"604769b0-503d-444f-983e-265898035e90\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " Nov 03 17:46:32.105658 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.105536 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-trusted-ca-bundle\") pod \"604769b0-503d-444f-983e-265898035e90\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " Nov 03 17:46:32.105658 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.105583 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"604769b0-503d-444f-983e-265898035e90\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " Nov 03 17:46:32.105658 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.105618 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-thanos-prometheus-http-client-file\") pod \"604769b0-503d-444f-983e-265898035e90\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " Nov 03 17:46:32.105658 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.105655 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-kubelet-serving-ca-bundle\") pod \"604769b0-503d-444f-983e-265898035e90\" (UID: \"604769b0-503d-444f-983e-265898035e90\") " Nov 03 17:46:32.106210 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.106188 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-kubelet-serving-ca-bundle" pod "604769b0-503d-444f-983e-265898035e90" (UID: "604769b0-503d-444f-983e-265898035e90"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:46:32.106594 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.106562 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-0" pod "604769b0-503d-444f-983e-265898035e90" (UID: "604769b0-503d-444f-983e-265898035e90"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:46:32.106920 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.106890 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-1" pod "604769b0-503d-444f-983e-265898035e90" (UID: "604769b0-503d-444f-983e-265898035e90"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:46:32.107741 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.107264 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-db" pod "604769b0-503d-444f-983e-265898035e90" (UID: "604769b0-503d-444f-983e-265898035e90"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:32.107826 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.107683 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-2" pod "604769b0-503d-444f-983e-265898035e90" (UID: "604769b0-503d-444f-983e-265898035e90"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:46:32.108141 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.108113 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-metrics-client-ca" pod "604769b0-503d-444f-983e-265898035e90" (UID: "604769b0-503d-444f-983e-265898035e90"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:46:32.108210 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.108123 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-trusted-ca-bundle" pod "604769b0-503d-444f-983e-265898035e90" (UID: "604769b0-503d-444f-983e-265898035e90"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:46:32.108447 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.108423 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-serving-certs-ca-bundle" pod "604769b0-503d-444f-983e-265898035e90" (UID: "604769b0-503d-444f-983e-265898035e90"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:46:32.108860 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.108814 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-config" pod "604769b0-503d-444f-983e-265898035e90" (UID: "604769b0-503d-444f-983e-265898035e90"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:46:32.108982 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.108885 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-tls" pod "604769b0-503d-444f-983e-265898035e90" (UID: "604769b0-503d-444f-983e-265898035e90"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:46:32.109043 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.108991 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-kube-rbac-proxy" pod "604769b0-503d-444f-983e-265898035e90" (UID: "604769b0-503d-444f-983e-265898035e90"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:46:32.109673 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.109641 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604769b0-503d-444f-983e-265898035e90-kube-api-access-xnmr6" pod "604769b0-503d-444f-983e-265898035e90" (UID: "604769b0-503d-444f-983e-265898035e90"). InnerVolumeSpecName "kube-api-access-xnmr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:46:32.110094 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.110067 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604769b0-503d-444f-983e-265898035e90-tls-assets" pod "604769b0-503d-444f-983e-265898035e90" (UID: "604769b0-503d-444f-983e-265898035e90"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:46:32.110614 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.110590 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-thanos-sidecar-tls" pod "604769b0-503d-444f-983e-265898035e90" (UID: "604769b0-503d-444f-983e-265898035e90"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:46:32.110715 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.110611 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/604769b0-503d-444f-983e-265898035e90-config-out" pod "604769b0-503d-444f-983e-265898035e90" (UID: "604769b0-503d-444f-983e-265898035e90"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:46:32.110914 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.110891 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-grpc-tls" pod "604769b0-503d-444f-983e-265898035e90" (UID: "604769b0-503d-444f-983e-265898035e90"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:46:32.110984 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.110928 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-thanos-prometheus-http-client-file" pod "604769b0-503d-444f-983e-265898035e90" (UID: "604769b0-503d-444f-983e-265898035e90"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:46:32.111058 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.111039 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-kube-rbac-proxy-web" pod "604769b0-503d-444f-983e-265898035e90" (UID: "604769b0-503d-444f-983e-265898035e90"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:46:32.111696 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.111680 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-metrics-client-certs" pod "604769b0-503d-444f-983e-265898035e90" (UID: "604769b0-503d-444f-983e-265898035e90"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:46:32.118744 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.118726 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-web-config" pod "604769b0-503d-444f-983e-265898035e90" (UID: "604769b0-503d-444f-983e-265898035e90"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:46:32.160525 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.160497 5805 generic.go:356] "Generic (PLEG): container finished" podID="604769b0-503d-444f-983e-265898035e90" containerID="85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac" exitCode=0 Nov 03 17:46:32.160525 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.160518 5805 generic.go:356] "Generic (PLEG): container finished" podID="604769b0-503d-444f-983e-265898035e90" containerID="83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341" exitCode=0 Nov 03 17:46:32.160525 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.160524 5805 generic.go:356] "Generic (PLEG): container finished" podID="604769b0-503d-444f-983e-265898035e90" containerID="ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488" exitCode=0 Nov 03 17:46:32.160525 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.160529 5805 generic.go:356] "Generic (PLEG): container finished" podID="604769b0-503d-444f-983e-265898035e90" containerID="54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f" exitCode=0 Nov 03 17:46:32.160920 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.160534 5805 generic.go:356] "Generic (PLEG): container finished" podID="604769b0-503d-444f-983e-265898035e90" containerID="78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562" exitCode=0 Nov 03 17:46:32.160920 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.160539 5805 generic.go:356] "Generic (PLEG): container finished" podID="604769b0-503d-444f-983e-265898035e90" containerID="5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2" exitCode=0 Nov 03 17:46:32.160920 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.160580 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.160920 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.160576 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"604769b0-503d-444f-983e-265898035e90","Type":"ContainerDied","Data":"85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac"} Nov 03 17:46:32.160920 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.160723 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"604769b0-503d-444f-983e-265898035e90","Type":"ContainerDied","Data":"83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341"} Nov 03 17:46:32.160920 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.160757 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"604769b0-503d-444f-983e-265898035e90","Type":"ContainerDied","Data":"ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488"} Nov 03 17:46:32.160920 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.160787 5805 scope.go:122] "RemoveContainer" containerID="85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac" Nov 03 17:46:32.160920 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.160795 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"604769b0-503d-444f-983e-265898035e90","Type":"ContainerDied","Data":"54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f"} Nov 03 17:46:32.160920 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.160819 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"604769b0-503d-444f-983e-265898035e90","Type":"ContainerDied","Data":"78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562"} Nov 03 17:46:32.160920 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.160845 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"604769b0-503d-444f-983e-265898035e90","Type":"ContainerDied","Data":"5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2"} Nov 03 17:46:32.160920 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.160870 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"604769b0-503d-444f-983e-265898035e90","Type":"ContainerDied","Data":"4d74568c685e0144ec9932191f629f76220da113cbdff3176f508cc6984c07c2"} Nov 03 17:46:32.168151 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.168083 5805 scope.go:122] "RemoveContainer" containerID="83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341" Nov 03 17:46:32.174747 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.174731 5805 scope.go:122] "RemoveContainer" containerID="ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488" Nov 03 17:46:32.180947 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.180932 5805 scope.go:122] "RemoveContainer" containerID="54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f" Nov 03 17:46:32.189514 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.189491 5805 scope.go:122] "RemoveContainer" containerID="78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562" Nov 03 17:46:32.197014 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.196990 5805 scope.go:122] "RemoveContainer" containerID="5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2" Nov 03 17:46:32.204464 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.204440 5805 scope.go:122] "RemoveContainer" containerID="15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2" Nov 03 17:46:32.207092 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.207062 5805 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-kube-rbac-proxy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:32.207092 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.207086 5805 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-db\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:32.207258 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.207098 5805 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:32.207258 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.207113 5805 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-config\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:32.207258 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.207126 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnmr6\" (UniqueName: \"kubernetes.io/projected/604769b0-503d-444f-983e-265898035e90-kube-api-access-xnmr6\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:32.207258 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.207142 5805 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:32.207258 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.207157 5805 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/604769b0-503d-444f-983e-265898035e90-tls-assets\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:32.207258 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.207170 5805 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-grpc-tls\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:32.207258 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.207183 5805 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-web-config\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:32.207258 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.207200 5805 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-metrics-client-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:32.207258 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.207214 5805 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-metrics-client-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:32.207258 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.207229 5805 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:32.207258 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.207245 5805 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/604769b0-503d-444f-983e-265898035e90-config-out\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:32.207258 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.207259 5805 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:32.207786 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.207275 5805 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:32.207786 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.207290 5805 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:32.207786 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.207306 5805 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:32.207786 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.207322 5805 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/604769b0-503d-444f-983e-265898035e90-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:32.207786 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.207339 5805 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:32.207786 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.207355 5805 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/604769b0-503d-444f-983e-265898035e90-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:46:32.211999 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.211983 5805 scope.go:122] "RemoveContainer" containerID="85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac" Nov 03 17:46:32.212234 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:32.212216 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac\": container with ID starting with 85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac not found: ID does not exist" containerID="85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac" Nov 03 17:46:32.212310 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.212246 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac"} err="failed to get container status \"85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac\": rpc error: code = NotFound desc = could not find container \"85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac\": container with ID starting with 85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac not found: ID does not exist" Nov 03 17:46:32.212310 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.212271 5805 scope.go:122] "RemoveContainer" containerID="83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341" Nov 03 17:46:32.212577 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:32.212559 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341\": container with ID starting with 83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341 not found: ID does not exist" containerID="83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341" Nov 03 17:46:32.212620 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.212585 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341"} err="failed to get container status \"83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341\": rpc error: code = NotFound desc = could not find container \"83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341\": container with ID starting with 83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341 not found: ID does not exist" Nov 03 17:46:32.212620 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.212601 5805 scope.go:122] "RemoveContainer" containerID="ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488" Nov 03 17:46:32.212836 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:32.212819 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488\": container with ID starting with ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488 not found: ID does not exist" containerID="ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488" Nov 03 17:46:32.212901 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.212840 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488"} err="failed to get container status \"ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488\": rpc error: code = NotFound desc = could not find container \"ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488\": container with ID starting with ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488 not found: ID does not exist" Nov 03 17:46:32.212901 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.212853 5805 scope.go:122] "RemoveContainer" containerID="54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f" Nov 03 17:46:32.213094 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:32.213076 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f\": container with ID starting with 54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f not found: ID does not exist" containerID="54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f" Nov 03 17:46:32.213151 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.213103 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f"} err="failed to get container status \"54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f\": rpc error: code = NotFound desc = could not find container \"54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f\": container with ID starting with 54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f not found: ID does not exist" Nov 03 17:46:32.213151 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.213125 5805 scope.go:122] "RemoveContainer" containerID="78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562" Nov 03 17:46:32.213349 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:32.213331 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562\": container with ID starting with 78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562 not found: ID does not exist" containerID="78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562" Nov 03 17:46:32.213420 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.213356 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562"} err="failed to get container status \"78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562\": rpc error: code = NotFound desc = could not find container \"78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562\": container with ID starting with 78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562 not found: ID does not exist" Nov 03 17:46:32.213420 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.213378 5805 scope.go:122] "RemoveContainer" containerID="5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2" Nov 03 17:46:32.213622 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:32.213607 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2\": container with ID starting with 5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2 not found: ID does not exist" containerID="5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2" Nov 03 17:46:32.213679 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.213628 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2"} err="failed to get container status \"5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2\": rpc error: code = NotFound desc = could not find container \"5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2\": container with ID starting with 5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2 not found: ID does not exist" Nov 03 17:46:32.213679 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.213647 5805 scope.go:122] "RemoveContainer" containerID="15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2" Nov 03 17:46:32.213921 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:32.213897 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2\": container with ID starting with 15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2 not found: ID does not exist" containerID="15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2" Nov 03 17:46:32.213968 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.213925 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2"} err="failed to get container status \"15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2\": rpc error: code = NotFound desc = could not find container \"15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2\": container with ID starting with 15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2 not found: ID does not exist" Nov 03 17:46:32.213968 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.213946 5805 scope.go:122] "RemoveContainer" containerID="85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac" Nov 03 17:46:32.214201 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.214182 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac"} err="failed to get container status \"85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac\": rpc error: code = NotFound desc = could not find container \"85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac\": container with ID starting with 85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac not found: ID does not exist" Nov 03 17:46:32.214272 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.214202 5805 scope.go:122] "RemoveContainer" containerID="83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341" Nov 03 17:46:32.214468 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.214452 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341"} err="failed to get container status \"83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341\": rpc error: code = NotFound desc = could not find container \"83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341\": container with ID starting with 83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341 not found: ID does not exist" Nov 03 17:46:32.214468 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.214467 5805 scope.go:122] "RemoveContainer" containerID="ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488" Nov 03 17:46:32.214703 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.214685 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488"} err="failed to get container status \"ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488\": rpc error: code = NotFound desc = could not find container \"ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488\": container with ID starting with ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488 not found: ID does not exist" Nov 03 17:46:32.214750 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.214703 5805 scope.go:122] "RemoveContainer" containerID="54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f" Nov 03 17:46:32.214967 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.214942 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f"} err="failed to get container status \"54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f\": rpc error: code = NotFound desc = could not find container \"54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f\": container with ID starting with 54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f not found: ID does not exist" Nov 03 17:46:32.215021 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.214970 5805 scope.go:122] "RemoveContainer" containerID="78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562" Nov 03 17:46:32.215272 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.215243 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562"} err="failed to get container status \"78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562\": rpc error: code = NotFound desc = could not find container \"78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562\": container with ID starting with 78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562 not found: ID does not exist" Nov 03 17:46:32.215272 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.215264 5805 scope.go:122] "RemoveContainer" containerID="5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2" Nov 03 17:46:32.215503 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.215486 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2"} err="failed to get container status \"5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2\": rpc error: code = NotFound desc = could not find container \"5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2\": container with ID starting with 5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2 not found: ID does not exist" Nov 03 17:46:32.215551 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.215503 5805 scope.go:122] "RemoveContainer" containerID="15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2" Nov 03 17:46:32.215694 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.215670 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2"} err="failed to get container status \"15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2\": rpc error: code = NotFound desc = could not find container \"15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2\": container with ID starting with 15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2 not found: ID does not exist" Nov 03 17:46:32.215734 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.215695 5805 scope.go:122] "RemoveContainer" containerID="85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac" Nov 03 17:46:32.215921 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.215904 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac"} err="failed to get container status \"85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac\": rpc error: code = NotFound desc = could not find container \"85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac\": container with ID starting with 85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac not found: ID does not exist" Nov 03 17:46:32.215967 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.215921 5805 scope.go:122] "RemoveContainer" containerID="83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341" Nov 03 17:46:32.216123 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.216107 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341"} err="failed to get container status \"83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341\": rpc error: code = NotFound desc = could not find container \"83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341\": container with ID starting with 83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341 not found: ID does not exist" Nov 03 17:46:32.216166 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.216122 5805 scope.go:122] "RemoveContainer" containerID="ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488" Nov 03 17:46:32.216387 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.216358 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488"} err="failed to get container status \"ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488\": rpc error: code = NotFound desc = could not find container \"ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488\": container with ID starting with ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488 not found: ID does not exist" Nov 03 17:46:32.216387 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.216379 5805 scope.go:122] "RemoveContainer" containerID="54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f" Nov 03 17:46:32.216686 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.216664 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f"} err="failed to get container status \"54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f\": rpc error: code = NotFound desc = could not find container \"54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f\": container with ID starting with 54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f not found: ID does not exist" Nov 03 17:46:32.216686 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.216683 5805 scope.go:122] "RemoveContainer" containerID="78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562" Nov 03 17:46:32.216933 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.216915 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562"} err="failed to get container status \"78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562\": rpc error: code = NotFound desc = could not find container \"78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562\": container with ID starting with 78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562 not found: ID does not exist" Nov 03 17:46:32.216998 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.216934 5805 scope.go:122] "RemoveContainer" containerID="5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2" Nov 03 17:46:32.217166 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.217146 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2"} err="failed to get container status \"5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2\": rpc error: code = NotFound desc = could not find container \"5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2\": container with ID starting with 5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2 not found: ID does not exist" Nov 03 17:46:32.217166 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.217163 5805 scope.go:122] "RemoveContainer" containerID="15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2" Nov 03 17:46:32.217446 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.217424 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2"} err="failed to get container status \"15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2\": rpc error: code = NotFound desc = could not find container \"15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2\": container with ID starting with 15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2 not found: ID does not exist" Nov 03 17:46:32.217446 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.217444 5805 scope.go:122] "RemoveContainer" containerID="85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac" Nov 03 17:46:32.217715 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.217697 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac"} err="failed to get container status \"85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac\": rpc error: code = NotFound desc = could not find container \"85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac\": container with ID starting with 85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac not found: ID does not exist" Nov 03 17:46:32.217776 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.217715 5805 scope.go:122] "RemoveContainer" containerID="83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341" Nov 03 17:46:32.217979 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.217960 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341"} err="failed to get container status \"83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341\": rpc error: code = NotFound desc = could not find container \"83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341\": container with ID starting with 83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341 not found: ID does not exist" Nov 03 17:46:32.218033 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.217979 5805 scope.go:122] "RemoveContainer" containerID="ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488" Nov 03 17:46:32.218222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.218199 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488"} err="failed to get container status \"ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488\": rpc error: code = NotFound desc = could not find container \"ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488\": container with ID starting with ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488 not found: ID does not exist" Nov 03 17:46:32.218222 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.218219 5805 scope.go:122] "RemoveContainer" containerID="54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f" Nov 03 17:46:32.218498 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.218479 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f"} err="failed to get container status \"54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f\": rpc error: code = NotFound desc = could not find container \"54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f\": container with ID starting with 54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f not found: ID does not exist" Nov 03 17:46:32.218498 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.218497 5805 scope.go:122] "RemoveContainer" containerID="78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562" Nov 03 17:46:32.218735 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.218714 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562"} err="failed to get container status \"78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562\": rpc error: code = NotFound desc = could not find container \"78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562\": container with ID starting with 78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562 not found: ID does not exist" Nov 03 17:46:32.218780 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.218735 5805 scope.go:122] "RemoveContainer" containerID="5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2" Nov 03 17:46:32.218959 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.218942 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2"} err="failed to get container status \"5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2\": rpc error: code = NotFound desc = could not find container \"5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2\": container with ID starting with 5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2 not found: ID does not exist" Nov 03 17:46:32.219007 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.218959 5805 scope.go:122] "RemoveContainer" containerID="15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2" Nov 03 17:46:32.219171 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.219147 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2"} err="failed to get container status \"15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2\": rpc error: code = NotFound desc = could not find container \"15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2\": container with ID starting with 15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2 not found: ID does not exist" Nov 03 17:46:32.219215 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.219170 5805 scope.go:122] "RemoveContainer" containerID="85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac" Nov 03 17:46:32.219406 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.219381 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac"} err="failed to get container status \"85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac\": rpc error: code = NotFound desc = could not find container \"85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac\": container with ID starting with 85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac not found: ID does not exist" Nov 03 17:46:32.219493 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.219482 5805 scope.go:122] "RemoveContainer" containerID="83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341" Nov 03 17:46:32.219713 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.219692 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341"} err="failed to get container status \"83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341\": rpc error: code = NotFound desc = could not find container \"83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341\": container with ID starting with 83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341 not found: ID does not exist" Nov 03 17:46:32.219713 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.219711 5805 scope.go:122] "RemoveContainer" containerID="ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488" Nov 03 17:46:32.219956 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.219937 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488"} err="failed to get container status \"ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488\": rpc error: code = NotFound desc = could not find container \"ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488\": container with ID starting with ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488 not found: ID does not exist" Nov 03 17:46:32.220026 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.219956 5805 scope.go:122] "RemoveContainer" containerID="54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f" Nov 03 17:46:32.220202 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.220184 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f"} err="failed to get container status \"54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f\": rpc error: code = NotFound desc = could not find container \"54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f\": container with ID starting with 54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f not found: ID does not exist" Nov 03 17:46:32.220265 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.220202 5805 scope.go:122] "RemoveContainer" containerID="78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562" Nov 03 17:46:32.220458 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.220441 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562"} err="failed to get container status \"78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562\": rpc error: code = NotFound desc = could not find container \"78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562\": container with ID starting with 78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562 not found: ID does not exist" Nov 03 17:46:32.220533 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.220459 5805 scope.go:122] "RemoveContainer" containerID="5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2" Nov 03 17:46:32.220742 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.220712 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2"} err="failed to get container status \"5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2\": rpc error: code = NotFound desc = could not find container \"5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2\": container with ID starting with 5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2 not found: ID does not exist" Nov 03 17:46:32.220804 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.220744 5805 scope.go:122] "RemoveContainer" containerID="15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2" Nov 03 17:46:32.220988 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.220965 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2"} err="failed to get container status \"15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2\": rpc error: code = NotFound desc = could not find container \"15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2\": container with ID starting with 15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2 not found: ID does not exist" Nov 03 17:46:32.221046 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.220988 5805 scope.go:122] "RemoveContainer" containerID="85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac" Nov 03 17:46:32.221204 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.221188 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac"} err="failed to get container status \"85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac\": rpc error: code = NotFound desc = could not find container \"85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac\": container with ID starting with 85017b5a6281f19b66ee6e8d386d7ebbaac72269da5dd7d9981eb88b3c4392ac not found: ID does not exist" Nov 03 17:46:32.221204 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.221205 5805 scope.go:122] "RemoveContainer" containerID="83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341" Nov 03 17:46:32.221420 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.221387 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341"} err="failed to get container status \"83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341\": rpc error: code = NotFound desc = could not find container \"83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341\": container with ID starting with 83742d6808cb966b3c44f4049443030c3fdf89d8aa22c93d7136952d9981e341 not found: ID does not exist" Nov 03 17:46:32.221420 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.221419 5805 scope.go:122] "RemoveContainer" containerID="ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488" Nov 03 17:46:32.221673 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.221656 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488"} err="failed to get container status \"ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488\": rpc error: code = NotFound desc = could not find container \"ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488\": container with ID starting with ed474bc0355187c9524ace734da31b037fcb137c31af1abdf1f036c435a84488 not found: ID does not exist" Nov 03 17:46:32.221673 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.221672 5805 scope.go:122] "RemoveContainer" containerID="54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f" Nov 03 17:46:32.221965 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.221947 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f"} err="failed to get container status \"54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f\": rpc error: code = NotFound desc = could not find container \"54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f\": container with ID starting with 54d0d732b4f28f878b960e8e8749d7797d05eb3d0590592d9786abf61903344f not found: ID does not exist" Nov 03 17:46:32.222038 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.221965 5805 scope.go:122] "RemoveContainer" containerID="78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562" Nov 03 17:46:32.222219 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.222199 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562"} err="failed to get container status \"78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562\": rpc error: code = NotFound desc = could not find container \"78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562\": container with ID starting with 78c34a2bb6e78783390b29dcc667f8d3d52cc9acfe677e4d4bfacd0f11975562 not found: ID does not exist" Nov 03 17:46:32.222289 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.222220 5805 scope.go:122] "RemoveContainer" containerID="5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2" Nov 03 17:46:32.222499 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.222465 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2"} err="failed to get container status \"5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2\": rpc error: code = NotFound desc = could not find container \"5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2\": container with ID starting with 5dd48930157ff6f5a8f170e432d1e6dd5615b0f6c8dd34b616e24557cd698ce2 not found: ID does not exist" Nov 03 17:46:32.222499 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.222489 5805 scope.go:122] "RemoveContainer" containerID="15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2" Nov 03 17:46:32.222717 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.222703 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2"} err="failed to get container status \"15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2\": rpc error: code = NotFound desc = could not find container \"15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2\": container with ID starting with 15aa59b386450b6984805665ae7b1e3ecb59a094abcbb51d431e493d0a27a6b2 not found: ID does not exist" Nov 03 17:46:32.288059 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.287987 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 17:46:32.302012 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.301965 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 17:46:32.333936 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.333912 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 17:46:32.334167 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.334156 5805 cpu_manager.go:414] "containerMap: removing container" podUID="604769b0-503d-444f-983e-265898035e90" containerName="init-config-reloader" Nov 03 17:46:32.334216 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.334168 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604769b0-503d-444f-983e-265898035e90" containerName="init-config-reloader" Nov 03 17:46:32.334216 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.334179 5805 cpu_manager.go:414] "containerMap: removing container" podUID="604769b0-503d-444f-983e-265898035e90" containerName="kube-rbac-proxy-thanos" Nov 03 17:46:32.334216 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.334184 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604769b0-503d-444f-983e-265898035e90" containerName="kube-rbac-proxy-thanos" Nov 03 17:46:32.334216 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.334195 5805 cpu_manager.go:414] "containerMap: removing container" podUID="604769b0-503d-444f-983e-265898035e90" containerName="kube-rbac-proxy" Nov 03 17:46:32.334216 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.334200 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604769b0-503d-444f-983e-265898035e90" containerName="kube-rbac-proxy" Nov 03 17:46:32.334216 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.334210 5805 cpu_manager.go:414] "containerMap: removing container" podUID="604769b0-503d-444f-983e-265898035e90" containerName="thanos-sidecar" Nov 03 17:46:32.334216 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.334214 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604769b0-503d-444f-983e-265898035e90" containerName="thanos-sidecar" Nov 03 17:46:32.334478 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.334222 5805 cpu_manager.go:414] "containerMap: removing container" podUID="604769b0-503d-444f-983e-265898035e90" containerName="config-reloader" Nov 03 17:46:32.334478 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.334226 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604769b0-503d-444f-983e-265898035e90" containerName="config-reloader" Nov 03 17:46:32.334478 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.334239 5805 cpu_manager.go:414] "containerMap: removing container" podUID="604769b0-503d-444f-983e-265898035e90" containerName="prometheus" Nov 03 17:46:32.334478 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.334243 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604769b0-503d-444f-983e-265898035e90" containerName="prometheus" Nov 03 17:46:32.334478 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.334253 5805 cpu_manager.go:414] "containerMap: removing container" podUID="604769b0-503d-444f-983e-265898035e90" containerName="kube-rbac-proxy-web" Nov 03 17:46:32.334478 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.334258 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604769b0-503d-444f-983e-265898035e90" containerName="kube-rbac-proxy-web" Nov 03 17:46:32.334478 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.334296 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="604769b0-503d-444f-983e-265898035e90" containerName="thanos-sidecar" Nov 03 17:46:32.334478 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.334310 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="604769b0-503d-444f-983e-265898035e90" containerName="config-reloader" Nov 03 17:46:32.334478 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.334319 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="604769b0-503d-444f-983e-265898035e90" containerName="prometheus" Nov 03 17:46:32.334478 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.334325 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="604769b0-503d-444f-983e-265898035e90" containerName="kube-rbac-proxy-web" Nov 03 17:46:32.334478 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.334330 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="604769b0-503d-444f-983e-265898035e90" containerName="init-config-reloader" Nov 03 17:46:32.334478 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.334335 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="604769b0-503d-444f-983e-265898035e90" containerName="kube-rbac-proxy-thanos" Nov 03 17:46:32.334478 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.334339 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="604769b0-503d-444f-983e-265898035e90" containerName="kube-rbac-proxy" Nov 03 17:46:32.368767 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.368741 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.407330 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.407306 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 17:46:32.408833 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.408809 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5440d502-9859-436b-bba4-acb92ec3282e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.408941 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.408856 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.408941 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.408922 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.409033 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.408991 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5440d502-9859-436b-bba4-acb92ec3282e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.409082 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.409031 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.409082 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.409046 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5440d502-9859-436b-bba4-acb92ec3282e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.409181 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.409096 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-config\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.409181 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.409136 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.409181 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.409163 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-web-config\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.409311 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.409187 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5440d502-9859-436b-bba4-acb92ec3282e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.409311 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.409247 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5440d502-9859-436b-bba4-acb92ec3282e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.409311 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.409271 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhncv\" (UniqueName: \"kubernetes.io/projected/5440d502-9859-436b-bba4-acb92ec3282e-kube-api-access-bhncv\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.409432 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.409324 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.409432 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.409364 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.409432 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.409405 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5440d502-9859-436b-bba4-acb92ec3282e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.409525 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.409436 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5440d502-9859-436b-bba4-acb92ec3282e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.409525 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.409463 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5440d502-9859-436b-bba4-acb92ec3282e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.409525 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.409507 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5440d502-9859-436b-bba4-acb92ec3282e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.409636 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.409527 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.409636 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.409546 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5440d502-9859-436b-bba4-acb92ec3282e-config-out\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.418510 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.418488 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7d5fs\"" Nov 03 17:46:32.418625 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.418525 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 17:46:32.418625 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.418528 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 17:46:32.418849 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.418833 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 17:46:32.418912 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.418898 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 17:46:32.418912 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.418903 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 17:46:32.418996 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.418921 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 17:46:32.419180 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.419164 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 17:46:32.419234 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.419184 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 17:46:32.419234 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.419199 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 17:46:32.419410 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.419385 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 17:46:32.419515 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.419499 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 17:46:32.419637 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.419620 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-251ja618v50s1\"" Nov 03 17:46:32.419748 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.419733 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 17:46:32.465359 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.465339 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 17:46:32.465450 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.465384 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 17:46:32.509944 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.509919 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5440d502-9859-436b-bba4-acb92ec3282e-config-out\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.510060 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.509948 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5440d502-9859-436b-bba4-acb92ec3282e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.510060 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.509972 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.510060 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.509988 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.510060 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.510009 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5440d502-9859-436b-bba4-acb92ec3282e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.510060 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.510034 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.510060 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.510057 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5440d502-9859-436b-bba4-acb92ec3282e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.510382 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.510084 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-config\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.510382 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.510111 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.510382 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.510137 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-web-config\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.510382 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.510162 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5440d502-9859-436b-bba4-acb92ec3282e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.510382 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.510193 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5440d502-9859-436b-bba4-acb92ec3282e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.510382 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.510217 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhncv\" (UniqueName: \"kubernetes.io/projected/5440d502-9859-436b-bba4-acb92ec3282e-kube-api-access-bhncv\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.510382 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.510240 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.510382 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.510265 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.510382 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.510282 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5440d502-9859-436b-bba4-acb92ec3282e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.510382 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.510304 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5440d502-9859-436b-bba4-acb92ec3282e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.510382 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.510326 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5440d502-9859-436b-bba4-acb92ec3282e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.510382 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.510345 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5440d502-9859-436b-bba4-acb92ec3282e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.511656 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.511435 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5440d502-9859-436b-bba4-acb92ec3282e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.512955 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.512929 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5440d502-9859-436b-bba4-acb92ec3282e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.513066 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.513044 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5440d502-9859-436b-bba4-acb92ec3282e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.513168 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.513149 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5440d502-9859-436b-bba4-acb92ec3282e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.513265 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.513235 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5440d502-9859-436b-bba4-acb92ec3282e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.513574 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.513491 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5440d502-9859-436b-bba4-acb92ec3282e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.513574 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.510372 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.513574 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.513534 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5440d502-9859-436b-bba4-acb92ec3282e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.514035 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.514006 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.514707 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.514683 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5440d502-9859-436b-bba4-acb92ec3282e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.515115 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.515094 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-web-config\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.515115 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.515093 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.515368 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.515341 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5440d502-9859-436b-bba4-acb92ec3282e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.515477 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.515435 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5440d502-9859-436b-bba4-acb92ec3282e-config-out\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.515561 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.515537 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.515624 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.515605 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.515957 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.515933 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.516630 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.516607 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.516711 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.516662 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.516907 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.516885 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5440d502-9859-436b-bba4-acb92ec3282e-config\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.570798 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.570772 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhncv\" (UniqueName: \"kubernetes.io/projected/5440d502-9859-436b-bba4-acb92ec3282e-kube-api-access-bhncv\") pod \"prometheus-k8s-0\" (UID: \"5440d502-9859-436b-bba4-acb92ec3282e\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.769171 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.769139 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:32.935415 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.935369 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8p8q8" Nov 03 17:46:32.974254 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:32.974223 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="604769b0-503d-444f-983e-265898035e90" path="/var/lib/kubelet/pods/604769b0-503d-444f-983e-265898035e90/volumes" Nov 03 17:46:33.225640 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:33.225564 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 17:46:33.233430 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:46:33.233389 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5440d502_9859_436b_bba4_acb92ec3282e.slice/crio-94f37e29776ec6cb4bed0ea6dd0c2055866cfd9e11d06afed1bae34e091999f0 WatchSource:0}: Error finding container 94f37e29776ec6cb4bed0ea6dd0c2055866cfd9e11d06afed1bae34e091999f0: Status 404 returned error can't find the container with id 94f37e29776ec6cb4bed0ea6dd0c2055866cfd9e11d06afed1bae34e091999f0 Nov 03 17:46:34.130023 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:34.129992 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:34.170322 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:34.170285 5805 generic.go:356] "Generic (PLEG): container finished" podID="5440d502-9859-436b-bba4-acb92ec3282e" containerID="f2aee62296400df15f6966a4f808738d9fb5f9c6309ba3e0db6b61c20113be55" exitCode=0 Nov 03 17:46:34.170465 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:34.170367 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5440d502-9859-436b-bba4-acb92ec3282e","Type":"ContainerDied","Data":"f2aee62296400df15f6966a4f808738d9fb5f9c6309ba3e0db6b61c20113be55"} Nov 03 17:46:34.170465 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:34.170450 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5440d502-9859-436b-bba4-acb92ec3282e","Type":"ContainerStarted","Data":"94f37e29776ec6cb4bed0ea6dd0c2055866cfd9e11d06afed1bae34e091999f0"} Nov 03 17:46:34.343867 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:34.343837 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:35.177091 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:35.177059 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5440d502-9859-436b-bba4-acb92ec3282e","Type":"ContainerStarted","Data":"4e8e3a90de33f3212c22fc9e693311f239948107cad95d283fec9b1310d09b35"} Nov 03 17:46:35.177240 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:35.177122 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5440d502-9859-436b-bba4-acb92ec3282e","Type":"ContainerStarted","Data":"16d00b2c3f44fd780305f967778fcd91176b647f8bbc168dead6922b60cfd96b"} Nov 03 17:46:35.177240 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:35.177147 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5440d502-9859-436b-bba4-acb92ec3282e","Type":"ContainerStarted","Data":"e5dc16f5bf3ebbb69feadd7a403485596b287fb077562a9405cca8516c4c15b4"} Nov 03 17:46:35.177240 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:35.177169 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5440d502-9859-436b-bba4-acb92ec3282e","Type":"ContainerStarted","Data":"5ff54fcefbbb4661d06889dd4c07f788964ca52a533747e433a8f35f69e9e4d9"} Nov 03 17:46:35.177240 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:35.177193 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5440d502-9859-436b-bba4-acb92ec3282e","Type":"ContainerStarted","Data":"37d76949999db84d53fc33a3eeae8d86d4682bc9209d0312b4919309734a06a3"} Nov 03 17:46:35.177240 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:35.177214 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5440d502-9859-436b-bba4-acb92ec3282e","Type":"ContainerStarted","Data":"4ccde71432690fda89b3b1b20a3501c84a8d532982c7201db0896f21541dd152"} Nov 03 17:46:35.306891 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:35.306838 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.287803873 podStartE2EDuration="3.306823249s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.019376ms" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:35.305194623 +0000 UTC m=+32.438823634" watchObservedRunningTime="2026-11-03 17:46:35.306823249 +0000 UTC m=+32.440452249" Nov 03 17:46:36.848409 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:36.848360 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:37.059174 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:37.059139 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:46:37.769581 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:46:37.769546 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:46:56.233891 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:46:56.233853 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:47:05.111473 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:47:05.111440 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:47:05.352339 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:47:05.352294 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:47:19.892251 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:47:19.892211 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:47:19.907608 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:47:19.907582 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:47:20.344118 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:47:20.344092 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 17:47:26.234594 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:47:26.234560 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:47:56.233429 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:47:56.233333 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:48:05.452660 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:05.452624 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:48:14.849109 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:14.849071 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:48:15.060387 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:15.060353 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:48:23.227223 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:23.227184 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:48:26.234385 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:48:26.234344 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:48:31.643846 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:31.643813 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:48:50.886412 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:50.886359 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:48:51.151111 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:51.151032 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:48:52.959624 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:52.959588 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm"] Nov 03 17:48:52.962189 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:52.962168 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm" Nov 03 17:48:53.048237 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:53.048200 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:48:53.060958 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:53.060933 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-s42jz\"" Nov 03 17:48:53.060958 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:53.060953 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 17:48:53.061096 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:53.060984 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 17:48:53.061656 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:53.061638 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm"] Nov 03 17:48:53.096055 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:53.096022 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2rr5\" (UniqueName: \"kubernetes.io/projected/1ebebdbe-ef52-4178-8007-432690ec1dc0-kube-api-access-g2rr5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm\" (UID: \"1ebebdbe-ef52-4178-8007-432690ec1dc0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm" Nov 03 17:48:53.096055 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:53.096057 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ebebdbe-ef52-4178-8007-432690ec1dc0-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm\" (UID: \"1ebebdbe-ef52-4178-8007-432690ec1dc0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm" Nov 03 17:48:53.096247 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:53.096096 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ebebdbe-ef52-4178-8007-432690ec1dc0-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm\" (UID: \"1ebebdbe-ef52-4178-8007-432690ec1dc0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm" Nov 03 17:48:53.197127 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:53.197092 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2rr5\" (UniqueName: \"kubernetes.io/projected/1ebebdbe-ef52-4178-8007-432690ec1dc0-kube-api-access-g2rr5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm\" (UID: \"1ebebdbe-ef52-4178-8007-432690ec1dc0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm" Nov 03 17:48:53.197323 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:53.197146 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ebebdbe-ef52-4178-8007-432690ec1dc0-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm\" (UID: \"1ebebdbe-ef52-4178-8007-432690ec1dc0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm" Nov 03 17:48:53.197323 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:53.197176 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ebebdbe-ef52-4178-8007-432690ec1dc0-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm\" (UID: \"1ebebdbe-ef52-4178-8007-432690ec1dc0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm" Nov 03 17:48:53.197637 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:53.197618 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ebebdbe-ef52-4178-8007-432690ec1dc0-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm\" (UID: \"1ebebdbe-ef52-4178-8007-432690ec1dc0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm" Nov 03 17:48:53.197822 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:53.197802 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ebebdbe-ef52-4178-8007-432690ec1dc0-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm\" (UID: \"1ebebdbe-ef52-4178-8007-432690ec1dc0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm" Nov 03 17:48:53.265119 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:53.265050 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2rr5\" (UniqueName: \"kubernetes.io/projected/1ebebdbe-ef52-4178-8007-432690ec1dc0-kube-api-access-g2rr5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm\" (UID: \"1ebebdbe-ef52-4178-8007-432690ec1dc0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm" Nov 03 17:48:53.275980 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:53.275955 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm" Nov 03 17:48:53.369423 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:53.369363 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:48:53.650000 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:53.649965 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm"] Nov 03 17:48:53.657922 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:48:53.657897 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ebebdbe_ef52_4178_8007_432690ec1dc0.slice/crio-26db43168992037991956ad4ff46942008f2f2548499b90839783567a6740e6e WatchSource:0}: Error finding container 26db43168992037991956ad4ff46942008f2f2548499b90839783567a6740e6e: Status 404 returned error can't find the container with id 26db43168992037991956ad4ff46942008f2f2548499b90839783567a6740e6e Nov 03 17:48:53.659854 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:53.659832 5805 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 17:48:53.809680 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:53.809646 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9"] Nov 03 17:48:53.813133 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:53.813101 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9" Nov 03 17:48:53.862112 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:53.862088 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-92cwp\"" Nov 03 17:48:53.870259 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:53.870233 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9"] Nov 03 17:48:54.004186 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:54.004101 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9bgv\" (UniqueName: \"kubernetes.io/projected/6dffb926-bfdc-47b8-aaab-b3cb58178091-kube-api-access-f9bgv\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9\" (UID: \"6dffb926-bfdc-47b8-aaab-b3cb58178091\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9" Nov 03 17:48:54.004186 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:54.004169 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6dffb926-bfdc-47b8-aaab-b3cb58178091-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9\" (UID: \"6dffb926-bfdc-47b8-aaab-b3cb58178091\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9" Nov 03 17:48:54.004627 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:54.004230 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6dffb926-bfdc-47b8-aaab-b3cb58178091-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9\" (UID: \"6dffb926-bfdc-47b8-aaab-b3cb58178091\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9" Nov 03 17:48:54.104981 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:54.104946 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6dffb926-bfdc-47b8-aaab-b3cb58178091-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9\" (UID: \"6dffb926-bfdc-47b8-aaab-b3cb58178091\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9" Nov 03 17:48:54.105158 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:54.104993 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6dffb926-bfdc-47b8-aaab-b3cb58178091-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9\" (UID: \"6dffb926-bfdc-47b8-aaab-b3cb58178091\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9" Nov 03 17:48:54.105158 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:54.105035 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9bgv\" (UniqueName: \"kubernetes.io/projected/6dffb926-bfdc-47b8-aaab-b3cb58178091-kube-api-access-f9bgv\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9\" (UID: \"6dffb926-bfdc-47b8-aaab-b3cb58178091\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9" Nov 03 17:48:54.105433 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:54.105389 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6dffb926-bfdc-47b8-aaab-b3cb58178091-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9\" (UID: \"6dffb926-bfdc-47b8-aaab-b3cb58178091\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9" Nov 03 17:48:54.105493 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:54.105426 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6dffb926-bfdc-47b8-aaab-b3cb58178091-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9\" (UID: \"6dffb926-bfdc-47b8-aaab-b3cb58178091\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9" Nov 03 17:48:54.162741 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:54.162712 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9bgv\" (UniqueName: \"kubernetes.io/projected/6dffb926-bfdc-47b8-aaab-b3cb58178091-kube-api-access-f9bgv\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9\" (UID: \"6dffb926-bfdc-47b8-aaab-b3cb58178091\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9" Nov 03 17:48:54.427476 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:54.427442 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9" Nov 03 17:48:54.611101 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:54.611059 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:48:54.643958 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:54.643924 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm" event={"ID":"1ebebdbe-ef52-4178-8007-432690ec1dc0","Type":"ContainerStarted","Data":"26db43168992037991956ad4ff46942008f2f2548499b90839783567a6740e6e"} Nov 03 17:48:54.796603 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:54.796523 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9"] Nov 03 17:48:54.803862 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:48:54.803829 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dffb926_bfdc_47b8_aaab_b3cb58178091.slice/crio-5e8634705a070377c50747b57aaba2da12b56dcde78ee09f7968e90c198bdc63 WatchSource:0}: Error finding container 5e8634705a070377c50747b57aaba2da12b56dcde78ee09f7968e90c198bdc63: Status 404 returned error can't find the container with id 5e8634705a070377c50747b57aaba2da12b56dcde78ee09f7968e90c198bdc63 Nov 03 17:48:55.648894 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:55.648864 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9" event={"ID":"6dffb926-bfdc-47b8-aaab-b3cb58178091","Type":"ContainerStarted","Data":"5e8634705a070377c50747b57aaba2da12b56dcde78ee09f7968e90c198bdc63"} Nov 03 17:48:56.242089 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:48:56.242053 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:48:57.503078 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.503038 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87"] Nov 03 17:48:57.506108 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.506086 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87" Nov 03 17:48:57.534148 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.534122 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f73fa28-8822-499b-907a-3bbfc864780c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87\" (UID: \"8f73fa28-8822-499b-907a-3bbfc864780c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87" Nov 03 17:48:57.534292 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.534158 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbsvb\" (UniqueName: \"kubernetes.io/projected/8f73fa28-8822-499b-907a-3bbfc864780c-kube-api-access-xbsvb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87\" (UID: \"8f73fa28-8822-499b-907a-3bbfc864780c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87" Nov 03 17:48:57.534421 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.534344 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f73fa28-8822-499b-907a-3bbfc864780c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87\" (UID: \"8f73fa28-8822-499b-907a-3bbfc864780c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87" Nov 03 17:48:57.602081 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.602040 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-sg8qv\"" Nov 03 17:48:57.602852 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.602822 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87"] Nov 03 17:48:57.606680 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.606659 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf"] Nov 03 17:48:57.609238 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.609224 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf" Nov 03 17:48:57.635136 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.635110 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f73fa28-8822-499b-907a-3bbfc864780c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87\" (UID: \"8f73fa28-8822-499b-907a-3bbfc864780c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87" Nov 03 17:48:57.635247 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.635143 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbsvb\" (UniqueName: \"kubernetes.io/projected/8f73fa28-8822-499b-907a-3bbfc864780c-kube-api-access-xbsvb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87\" (UID: \"8f73fa28-8822-499b-907a-3bbfc864780c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87" Nov 03 17:48:57.635247 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.635189 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a36b25d-d9ad-4209-818a-cb4b198ee862-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf\" (UID: \"0a36b25d-d9ad-4209-818a-cb4b198ee862\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf" Nov 03 17:48:57.635336 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.635252 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hht6\" (UniqueName: \"kubernetes.io/projected/0a36b25d-d9ad-4209-818a-cb4b198ee862-kube-api-access-9hht6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf\" (UID: \"0a36b25d-d9ad-4209-818a-cb4b198ee862\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf" Nov 03 17:48:57.635336 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.635314 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a36b25d-d9ad-4209-818a-cb4b198ee862-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf\" (UID: \"0a36b25d-d9ad-4209-818a-cb4b198ee862\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf" Nov 03 17:48:57.635440 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.635430 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f73fa28-8822-499b-907a-3bbfc864780c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87\" (UID: \"8f73fa28-8822-499b-907a-3bbfc864780c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87" Nov 03 17:48:57.635534 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.635518 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f73fa28-8822-499b-907a-3bbfc864780c-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87\" (UID: \"8f73fa28-8822-499b-907a-3bbfc864780c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87" Nov 03 17:48:57.635671 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.635658 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f73fa28-8822-499b-907a-3bbfc864780c-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87\" (UID: \"8f73fa28-8822-499b-907a-3bbfc864780c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87" Nov 03 17:48:57.658330 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.658308 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-rbqmd\"" Nov 03 17:48:57.670221 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.670195 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf"] Nov 03 17:48:57.694079 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.694054 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbsvb\" (UniqueName: \"kubernetes.io/projected/8f73fa28-8822-499b-907a-3bbfc864780c-kube-api-access-xbsvb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87\" (UID: \"8f73fa28-8822-499b-907a-3bbfc864780c\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87" Nov 03 17:48:57.706787 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.706750 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l"] Nov 03 17:48:57.709826 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.709786 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l" Nov 03 17:48:57.736029 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.736004 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hht6\" (UniqueName: \"kubernetes.io/projected/0a36b25d-d9ad-4209-818a-cb4b198ee862-kube-api-access-9hht6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf\" (UID: \"0a36b25d-d9ad-4209-818a-cb4b198ee862\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf" Nov 03 17:48:57.736163 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.736066 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a36b25d-d9ad-4209-818a-cb4b198ee862-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf\" (UID: \"0a36b25d-d9ad-4209-818a-cb4b198ee862\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf" Nov 03 17:48:57.736163 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.736094 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/951df12e-8840-405b-b461-a4fe2cf59485-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l\" (UID: \"951df12e-8840-405b-b461-a4fe2cf59485\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l" Nov 03 17:48:57.736163 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.736125 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbpn6\" (UniqueName: \"kubernetes.io/projected/951df12e-8840-405b-b461-a4fe2cf59485-kube-api-access-cbpn6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l\" (UID: \"951df12e-8840-405b-b461-a4fe2cf59485\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l" Nov 03 17:48:57.736292 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.736249 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/951df12e-8840-405b-b461-a4fe2cf59485-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l\" (UID: \"951df12e-8840-405b-b461-a4fe2cf59485\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l" Nov 03 17:48:57.736334 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.736300 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a36b25d-d9ad-4209-818a-cb4b198ee862-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf\" (UID: \"0a36b25d-d9ad-4209-818a-cb4b198ee862\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf" Nov 03 17:48:57.736463 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.736446 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a36b25d-d9ad-4209-818a-cb4b198ee862-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf\" (UID: \"0a36b25d-d9ad-4209-818a-cb4b198ee862\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf" Nov 03 17:48:57.736609 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.736589 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a36b25d-d9ad-4209-818a-cb4b198ee862-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf\" (UID: \"0a36b25d-d9ad-4209-818a-cb4b198ee862\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf" Nov 03 17:48:57.759029 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.758971 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-75zw9\"" Nov 03 17:48:57.766380 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.766351 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l"] Nov 03 17:48:57.794043 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.794008 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hht6\" (UniqueName: \"kubernetes.io/projected/0a36b25d-d9ad-4209-818a-cb4b198ee862-kube-api-access-9hht6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf\" (UID: \"0a36b25d-d9ad-4209-818a-cb4b198ee862\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf" Nov 03 17:48:57.822982 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.822954 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87" Nov 03 17:48:57.837130 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.837100 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/951df12e-8840-405b-b461-a4fe2cf59485-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l\" (UID: \"951df12e-8840-405b-b461-a4fe2cf59485\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l" Nov 03 17:48:57.837326 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.837244 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/951df12e-8840-405b-b461-a4fe2cf59485-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l\" (UID: \"951df12e-8840-405b-b461-a4fe2cf59485\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l" Nov 03 17:48:57.837326 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.837283 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbpn6\" (UniqueName: \"kubernetes.io/projected/951df12e-8840-405b-b461-a4fe2cf59485-kube-api-access-cbpn6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l\" (UID: \"951df12e-8840-405b-b461-a4fe2cf59485\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l" Nov 03 17:48:57.837483 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.837458 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/951df12e-8840-405b-b461-a4fe2cf59485-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l\" (UID: \"951df12e-8840-405b-b461-a4fe2cf59485\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l" Nov 03 17:48:57.837778 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.837750 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/951df12e-8840-405b-b461-a4fe2cf59485-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l\" (UID: \"951df12e-8840-405b-b461-a4fe2cf59485\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l" Nov 03 17:48:57.900288 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.900258 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbpn6\" (UniqueName: \"kubernetes.io/projected/951df12e-8840-405b-b461-a4fe2cf59485-kube-api-access-cbpn6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l\" (UID: \"951df12e-8840-405b-b461-a4fe2cf59485\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l" Nov 03 17:48:57.924041 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:57.924007 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf" Nov 03 17:48:58.026066 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:58.025995 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l" Nov 03 17:48:58.198454 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:58.198416 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87"] Nov 03 17:48:58.205983 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:48:58.205945 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f73fa28_8822_499b_907a_3bbfc864780c.slice/crio-39c4213bdcb3955e6c5cf5585b672cdb8ce68950f5968e54926b39f4eb66963f WatchSource:0}: Error finding container 39c4213bdcb3955e6c5cf5585b672cdb8ce68950f5968e54926b39f4eb66963f: Status 404 returned error can't find the container with id 39c4213bdcb3955e6c5cf5585b672cdb8ce68950f5968e54926b39f4eb66963f Nov 03 17:48:58.316781 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:58.316750 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf"] Nov 03 17:48:58.323510 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:48:58.323477 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a36b25d_d9ad_4209_818a_cb4b198ee862.slice/crio-016cdee04741a78fe4d202fc4ca02aed8e8ceade72843980757f8021259e8dad WatchSource:0}: Error finding container 016cdee04741a78fe4d202fc4ca02aed8e8ceade72843980757f8021259e8dad: Status 404 returned error can't find the container with id 016cdee04741a78fe4d202fc4ca02aed8e8ceade72843980757f8021259e8dad Nov 03 17:48:58.395773 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:58.395721 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l"] Nov 03 17:48:58.403291 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:48:58.403251 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951df12e_8840_405b_b461_a4fe2cf59485.slice/crio-c5db01c03c8ff2fc5f3421e81a7ddd7200b3ef12091338423865c94196287d3f WatchSource:0}: Error finding container c5db01c03c8ff2fc5f3421e81a7ddd7200b3ef12091338423865c94196287d3f: Status 404 returned error can't find the container with id c5db01c03c8ff2fc5f3421e81a7ddd7200b3ef12091338423865c94196287d3f Nov 03 17:48:58.663278 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:58.663189 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf" event={"ID":"0a36b25d-d9ad-4209-818a-cb4b198ee862","Type":"ContainerStarted","Data":"016cdee04741a78fe4d202fc4ca02aed8e8ceade72843980757f8021259e8dad"} Nov 03 17:48:58.664750 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:58.664704 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l" event={"ID":"951df12e-8840-405b-b461-a4fe2cf59485","Type":"ContainerStarted","Data":"c5db01c03c8ff2fc5f3421e81a7ddd7200b3ef12091338423865c94196287d3f"} Nov 03 17:48:58.666285 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:58.666229 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87" event={"ID":"8f73fa28-8822-499b-907a-3bbfc864780c","Type":"ContainerStarted","Data":"39c4213bdcb3955e6c5cf5585b672cdb8ce68950f5968e54926b39f4eb66963f"} Nov 03 17:48:59.869146 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:48:59.869106 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:00.070965 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:00.070926 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:01.334911 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.334784 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:01.486976 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.486939 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d"] Nov 03 17:49:01.492065 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.490948 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.542889 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.542855 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 17:49:01.575317 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.574791 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdccc585-8c1c-451d-bacd-3e5933cc4b06-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.575317 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.574856 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdccc585-8c1c-451d-bacd-3e5933cc4b06-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.575317 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.574887 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdccc585-8c1c-451d-bacd-3e5933cc4b06-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.575317 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.574918 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx2zs\" (UniqueName: \"kubernetes.io/projected/bdccc585-8c1c-451d-bacd-3e5933cc4b06-kube-api-access-zx2zs\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.575317 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.574950 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdccc585-8c1c-451d-bacd-3e5933cc4b06-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.575317 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.574989 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdccc585-8c1c-451d-bacd-3e5933cc4b06-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.575317 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.575036 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdccc585-8c1c-451d-bacd-3e5933cc4b06-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.575317 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.575088 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdccc585-8c1c-451d-bacd-3e5933cc4b06-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.575317 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.575121 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdccc585-8c1c-451d-bacd-3e5933cc4b06-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.578528 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.577961 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-s9nlz\"" Nov 03 17:49:01.578528 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.578035 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 17:49:01.578528 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.578041 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 17:49:01.590534 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.590419 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d"] Nov 03 17:49:01.646756 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.646724 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:01.675768 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.675733 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdccc585-8c1c-451d-bacd-3e5933cc4b06-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.675942 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.675794 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdccc585-8c1c-451d-bacd-3e5933cc4b06-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.675942 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.675849 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdccc585-8c1c-451d-bacd-3e5933cc4b06-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.675942 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.675877 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdccc585-8c1c-451d-bacd-3e5933cc4b06-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.675942 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.675911 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdccc585-8c1c-451d-bacd-3e5933cc4b06-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.676173 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.675949 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdccc585-8c1c-451d-bacd-3e5933cc4b06-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.676173 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.675966 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdccc585-8c1c-451d-bacd-3e5933cc4b06-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.676173 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.675992 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx2zs\" (UniqueName: \"kubernetes.io/projected/bdccc585-8c1c-451d-bacd-3e5933cc4b06-kube-api-access-zx2zs\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.676173 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.676015 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdccc585-8c1c-451d-bacd-3e5933cc4b06-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.676452 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.676430 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdccc585-8c1c-451d-bacd-3e5933cc4b06-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.677258 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.677028 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdccc585-8c1c-451d-bacd-3e5933cc4b06-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.677487 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.677466 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdccc585-8c1c-451d-bacd-3e5933cc4b06-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.677731 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.677706 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdccc585-8c1c-451d-bacd-3e5933cc4b06-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.678065 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.678042 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdccc585-8c1c-451d-bacd-3e5933cc4b06-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.680854 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.680830 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdccc585-8c1c-451d-bacd-3e5933cc4b06-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.681080 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.681032 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdccc585-8c1c-451d-bacd-3e5933cc4b06-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.736314 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.736262 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdccc585-8c1c-451d-bacd-3e5933cc4b06-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.738914 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.738885 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx2zs\" (UniqueName: \"kubernetes.io/projected/bdccc585-8c1c-451d-bacd-3e5933cc4b06-kube-api-access-zx2zs\") pod \"openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d\" (UID: \"bdccc585-8c1c-451d-bacd-3e5933cc4b06\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.810420 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.809406 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:01.853861 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:01.853741 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:02.064834 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:02.064801 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:02.261665 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:02.261486 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d"] Nov 03 17:49:02.272983 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:49:02.272947 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdccc585_8c1c_451d_bacd_3e5933cc4b06.slice/crio-eb062f1e3059e987a5570d83264820e67b78e2430b557c8ffee468f448dc87ab WatchSource:0}: Error finding container eb062f1e3059e987a5570d83264820e67b78e2430b557c8ffee468f448dc87ab: Status 404 returned error can't find the container with id eb062f1e3059e987a5570d83264820e67b78e2430b557c8ffee468f448dc87ab Nov 03 17:49:02.692090 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:02.692055 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" event={"ID":"bdccc585-8c1c-451d-bacd-3e5933cc4b06","Type":"ContainerStarted","Data":"eb062f1e3059e987a5570d83264820e67b78e2430b557c8ffee468f448dc87ab"} Nov 03 17:49:03.698672 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:03.698628 5805 generic.go:356] "Generic (PLEG): container finished" podID="6dffb926-bfdc-47b8-aaab-b3cb58178091" containerID="ec6fd4f1578d6610b25a1aa069ff1b8b164110ae901a5cfcb10e8cf972dd07a5" exitCode=0 Nov 03 17:49:03.699150 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:03.698696 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9" event={"ID":"6dffb926-bfdc-47b8-aaab-b3cb58178091","Type":"ContainerDied","Data":"ec6fd4f1578d6610b25a1aa069ff1b8b164110ae901a5cfcb10e8cf972dd07a5"} Nov 03 17:49:05.500161 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:05.500128 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:05.712991 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:05.712956 5805 generic.go:356] "Generic (PLEG): container finished" podID="951df12e-8840-405b-b461-a4fe2cf59485" containerID="0b3397357b933a4402361707dd1828c64e643d338fa4a768364d01b43b0fce0d" exitCode=0 Nov 03 17:49:05.713157 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:05.713039 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l" event={"ID":"951df12e-8840-405b-b461-a4fe2cf59485","Type":"ContainerDied","Data":"0b3397357b933a4402361707dd1828c64e643d338fa4a768364d01b43b0fce0d"} Nov 03 17:49:05.715789 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:05.715653 5805 generic.go:356] "Generic (PLEG): container finished" podID="0a36b25d-d9ad-4209-818a-cb4b198ee862" containerID="bca471ea442f7d0a323414262e636e2df0395dce9a701c41b388854a2b70d845" exitCode=0 Nov 03 17:49:05.715789 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:05.715728 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf" event={"ID":"0a36b25d-d9ad-4209-818a-cb4b198ee862","Type":"ContainerDied","Data":"bca471ea442f7d0a323414262e636e2df0395dce9a701c41b388854a2b70d845"} Nov 03 17:49:05.717460 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:05.717433 5805 generic.go:356] "Generic (PLEG): container finished" podID="1ebebdbe-ef52-4178-8007-432690ec1dc0" containerID="2c52653b48d588b6d629cef3ce165deb16aa5a255f7a0303634a41a2f6356495" exitCode=0 Nov 03 17:49:05.718158 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:05.718133 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm" event={"ID":"1ebebdbe-ef52-4178-8007-432690ec1dc0","Type":"ContainerDied","Data":"2c52653b48d588b6d629cef3ce165deb16aa5a255f7a0303634a41a2f6356495"} Nov 03 17:49:05.815914 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:05.815881 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:07.728214 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:07.728177 5805 generic.go:356] "Generic (PLEG): container finished" podID="0a36b25d-d9ad-4209-818a-cb4b198ee862" containerID="c37ccee60683c5751ba5e84ad6737fcb9395f8075aade4cbae5db214f0e1cfd3" exitCode=0 Nov 03 17:49:07.728661 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:07.728264 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf" event={"ID":"0a36b25d-d9ad-4209-818a-cb4b198ee862","Type":"ContainerDied","Data":"c37ccee60683c5751ba5e84ad6737fcb9395f8075aade4cbae5db214f0e1cfd3"} Nov 03 17:49:07.730705 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:07.730685 5805 generic.go:356] "Generic (PLEG): container finished" podID="1ebebdbe-ef52-4178-8007-432690ec1dc0" containerID="c3bdab2c196b4905855771edab70dd80f101098515d9b56419762af6c8c6a1e2" exitCode=0 Nov 03 17:49:07.730787 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:07.730765 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm" event={"ID":"1ebebdbe-ef52-4178-8007-432690ec1dc0","Type":"ContainerDied","Data":"c3bdab2c196b4905855771edab70dd80f101098515d9b56419762af6c8c6a1e2"} Nov 03 17:49:07.733365 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:07.733347 5805 generic.go:356] "Generic (PLEG): container finished" podID="951df12e-8840-405b-b461-a4fe2cf59485" containerID="dc5cc01369cccc1e121265676f490585a3bac7acbf1d3e50ecb95275731aad34" exitCode=0 Nov 03 17:49:07.733474 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:07.733430 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l" event={"ID":"951df12e-8840-405b-b461-a4fe2cf59485","Type":"ContainerDied","Data":"dc5cc01369cccc1e121265676f490585a3bac7acbf1d3e50ecb95275731aad34"} Nov 03 17:49:07.735131 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:07.735048 5805 generic.go:356] "Generic (PLEG): container finished" podID="8f73fa28-8822-499b-907a-3bbfc864780c" containerID="2d789f3b47348a73375ed097d364c1e93cc208e41d6dbd982cf529faebc019b2" exitCode=0 Nov 03 17:49:07.735131 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:07.735091 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87" event={"ID":"8f73fa28-8822-499b-907a-3bbfc864780c","Type":"ContainerDied","Data":"2d789f3b47348a73375ed097d364c1e93cc208e41d6dbd982cf529faebc019b2"} Nov 03 17:49:07.806073 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:07.806045 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:49:07.806166 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:07.806107 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:49:07.806166 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:07.806136 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:49:08.742004 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:08.741968 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" event={"ID":"bdccc585-8c1c-451d-bacd-3e5933cc4b06","Type":"ContainerStarted","Data":"50120503aa49dc1515836ff16ea86b3e0c654ca0838d161408416e876fe4b805"} Nov 03 17:49:08.811725 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:08.811688 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:08.817792 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:08.817763 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:08.903811 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:08.903760 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" podStartSLOduration=2.374972513 podStartE2EDuration="7.903746024s" totalImagesPullingTime="5.528773511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:08.903627643 +0000 UTC m=+186.037256653" watchObservedRunningTime="2026-11-03 17:49:08.903746024 +0000 UTC m=+186.037375029" Nov 03 17:49:09.749077 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:09.748994 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:09.750189 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:09.750164 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d" Nov 03 17:49:10.756190 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:10.756150 5805 generic.go:356] "Generic (PLEG): container finished" podID="8f73fa28-8822-499b-907a-3bbfc864780c" containerID="fcfffae8d5925448e6ab98295e977b307d7dd52e98f45c21e7b5224ef2dbba7e" exitCode=0 Nov 03 17:49:10.756686 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:10.756260 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87" event={"ID":"8f73fa28-8822-499b-907a-3bbfc864780c","Type":"ContainerDied","Data":"fcfffae8d5925448e6ab98295e977b307d7dd52e98f45c21e7b5224ef2dbba7e"} Nov 03 17:49:17.799895 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:17.799859 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l" event={"ID":"951df12e-8840-405b-b461-a4fe2cf59485","Type":"ContainerStarted","Data":"dcb3270a2b4b949698d0a8a024400be26bf69732cc2d6c095632394ed7aed76b"} Nov 03 17:49:17.801937 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:17.801910 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87" event={"ID":"8f73fa28-8822-499b-907a-3bbfc864780c","Type":"ContainerStarted","Data":"95415c7423c19536cf5e62b5b7da7f5653b9082e9328d95842f6b1601e9601af"} Nov 03 17:49:17.803804 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:17.803778 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf" event={"ID":"0a36b25d-d9ad-4209-818a-cb4b198ee862","Type":"ContainerStarted","Data":"e50634f56afd969d61c70276fdd61f73e2c2bca92e10e6f241f0a139885d653d"} Nov 03 17:49:17.805435 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:17.805414 5805 generic.go:356] "Generic (PLEG): container finished" podID="1ebebdbe-ef52-4178-8007-432690ec1dc0" containerID="b810ae9128eaaeeed120a5361cee2693f0c69c595459c6267fd03247ac8571eb" exitCode=0 Nov 03 17:49:17.805516 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:17.805443 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm" event={"ID":"1ebebdbe-ef52-4178-8007-432690ec1dc0","Type":"ContainerDied","Data":"b810ae9128eaaeeed120a5361cee2693f0c69c595459c6267fd03247ac8571eb"} Nov 03 17:49:17.955668 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:17.955623 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l" podStartSLOduration=3.091009701 podStartE2EDuration="20.955608429s" totalImagesPullingTime="17.864598728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:48:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:17.954657338 +0000 UTC m=+195.088286368" watchObservedRunningTime="2026-11-03 17:49:17.955608429 +0000 UTC m=+195.089237460" Nov 03 17:49:18.065756 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:18.065700 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf" podStartSLOduration=3.168669676 podStartE2EDuration="21.06568583s" totalImagesPullingTime="17.897016154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:48:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:18.065085444 +0000 UTC m=+195.198714455" watchObservedRunningTime="2026-11-03 17:49:18.06568583 +0000 UTC m=+195.199314840" Nov 03 17:49:18.176372 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:18.176316 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87" podStartSLOduration=3.395844175 podStartE2EDuration="21.176301438s" totalImagesPullingTime="17.780457263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:48:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:18.175059845 +0000 UTC m=+195.308688866" watchObservedRunningTime="2026-11-03 17:49:18.176301438 +0000 UTC m=+195.309930447" Nov 03 17:49:18.811451 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:18.811418 5805 generic.go:356] "Generic (PLEG): container finished" podID="8f73fa28-8822-499b-907a-3bbfc864780c" containerID="95415c7423c19536cf5e62b5b7da7f5653b9082e9328d95842f6b1601e9601af" exitCode=0 Nov 03 17:49:18.811913 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:18.811507 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87" event={"ID":"8f73fa28-8822-499b-907a-3bbfc864780c","Type":"ContainerDied","Data":"95415c7423c19536cf5e62b5b7da7f5653b9082e9328d95842f6b1601e9601af"} Nov 03 17:49:18.813614 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:18.813590 5805 generic.go:356] "Generic (PLEG): container finished" podID="0a36b25d-d9ad-4209-818a-cb4b198ee862" containerID="e50634f56afd969d61c70276fdd61f73e2c2bca92e10e6f241f0a139885d653d" exitCode=0 Nov 03 17:49:18.813694 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:18.813677 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf" event={"ID":"0a36b25d-d9ad-4209-818a-cb4b198ee862","Type":"ContainerDied","Data":"e50634f56afd969d61c70276fdd61f73e2c2bca92e10e6f241f0a139885d653d"} Nov 03 17:49:18.815610 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:18.815589 5805 generic.go:356] "Generic (PLEG): container finished" podID="951df12e-8840-405b-b461-a4fe2cf59485" containerID="dcb3270a2b4b949698d0a8a024400be26bf69732cc2d6c095632394ed7aed76b" exitCode=0 Nov 03 17:49:18.815731 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:18.815666 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l" event={"ID":"951df12e-8840-405b-b461-a4fe2cf59485","Type":"ContainerDied","Data":"dcb3270a2b4b949698d0a8a024400be26bf69732cc2d6c095632394ed7aed76b"} Nov 03 17:49:18.936194 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:18.936169 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm" Nov 03 17:49:19.058538 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:19.058511 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ebebdbe-ef52-4178-8007-432690ec1dc0-util\" (UniqueName: \"kubernetes.io/empty-dir/1ebebdbe-ef52-4178-8007-432690ec1dc0-util\") pod \"1ebebdbe-ef52-4178-8007-432690ec1dc0\" (UID: \"1ebebdbe-ef52-4178-8007-432690ec1dc0\") " Nov 03 17:49:19.058689 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:19.058590 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ebebdbe-ef52-4178-8007-432690ec1dc0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ebebdbe-ef52-4178-8007-432690ec1dc0-bundle\") pod \"1ebebdbe-ef52-4178-8007-432690ec1dc0\" (UID: \"1ebebdbe-ef52-4178-8007-432690ec1dc0\") " Nov 03 17:49:19.058689 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:19.058682 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ebebdbe-ef52-4178-8007-432690ec1dc0-kube-api-access-g2rr5\" (UniqueName: \"kubernetes.io/projected/1ebebdbe-ef52-4178-8007-432690ec1dc0-kube-api-access-g2rr5\") pod \"1ebebdbe-ef52-4178-8007-432690ec1dc0\" (UID: \"1ebebdbe-ef52-4178-8007-432690ec1dc0\") " Nov 03 17:49:19.059147 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:19.059122 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebebdbe-ef52-4178-8007-432690ec1dc0-bundle" pod "1ebebdbe-ef52-4178-8007-432690ec1dc0" (UID: "1ebebdbe-ef52-4178-8007-432690ec1dc0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:19.060958 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:19.060931 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebebdbe-ef52-4178-8007-432690ec1dc0-kube-api-access-g2rr5" pod "1ebebdbe-ef52-4178-8007-432690ec1dc0" (UID: "1ebebdbe-ef52-4178-8007-432690ec1dc0"). InnerVolumeSpecName "kube-api-access-g2rr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:49:19.063853 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:19.063801 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebebdbe-ef52-4178-8007-432690ec1dc0-util" pod "1ebebdbe-ef52-4178-8007-432690ec1dc0" (UID: "1ebebdbe-ef52-4178-8007-432690ec1dc0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:19.159685 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:19.159652 5805 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1ebebdbe-ef52-4178-8007-432690ec1dc0-bundle\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:19.159685 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:19.159680 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2rr5\" (UniqueName: \"kubernetes.io/projected/1ebebdbe-ef52-4178-8007-432690ec1dc0-kube-api-access-g2rr5\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:19.159872 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:19.159693 5805 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1ebebdbe-ef52-4178-8007-432690ec1dc0-util\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:19.820772 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:19.820738 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm" event={"ID":"1ebebdbe-ef52-4178-8007-432690ec1dc0","Type":"ContainerDied","Data":"26db43168992037991956ad4ff46942008f2f2548499b90839783567a6740e6e"} Nov 03 17:49:19.821142 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:19.820790 5805 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26db43168992037991956ad4ff46942008f2f2548499b90839783567a6740e6e" Nov 03 17:49:19.821142 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:19.820809 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71j4pjm" Nov 03 17:49:19.950892 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:19.950865 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf" Nov 03 17:49:20.002746 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.002633 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l" Nov 03 17:49:20.004456 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.004440 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87" Nov 03 17:49:20.067331 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.067308 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a36b25d-d9ad-4209-818a-cb4b198ee862-kube-api-access-9hht6\" (UniqueName: \"kubernetes.io/projected/0a36b25d-d9ad-4209-818a-cb4b198ee862-kube-api-access-9hht6\") pod \"0a36b25d-d9ad-4209-818a-cb4b198ee862\" (UID: \"0a36b25d-d9ad-4209-818a-cb4b198ee862\") " Nov 03 17:49:20.067487 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.067374 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/951df12e-8840-405b-b461-a4fe2cf59485-bundle\" (UniqueName: \"kubernetes.io/empty-dir/951df12e-8840-405b-b461-a4fe2cf59485-bundle\") pod \"951df12e-8840-405b-b461-a4fe2cf59485\" (UID: \"951df12e-8840-405b-b461-a4fe2cf59485\") " Nov 03 17:49:20.067776 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.067671 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/951df12e-8840-405b-b461-a4fe2cf59485-kube-api-access-cbpn6\" (UniqueName: \"kubernetes.io/projected/951df12e-8840-405b-b461-a4fe2cf59485-kube-api-access-cbpn6\") pod \"951df12e-8840-405b-b461-a4fe2cf59485\" (UID: \"951df12e-8840-405b-b461-a4fe2cf59485\") " Nov 03 17:49:20.067776 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.067727 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f73fa28-8822-499b-907a-3bbfc864780c-util\" (UniqueName: \"kubernetes.io/empty-dir/8f73fa28-8822-499b-907a-3bbfc864780c-util\") pod \"8f73fa28-8822-499b-907a-3bbfc864780c\" (UID: \"8f73fa28-8822-499b-907a-3bbfc864780c\") " Nov 03 17:49:20.067974 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.067778 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f73fa28-8822-499b-907a-3bbfc864780c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f73fa28-8822-499b-907a-3bbfc864780c-bundle\") pod \"8f73fa28-8822-499b-907a-3bbfc864780c\" (UID: \"8f73fa28-8822-499b-907a-3bbfc864780c\") " Nov 03 17:49:20.067974 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.067820 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a36b25d-d9ad-4209-818a-cb4b198ee862-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a36b25d-d9ad-4209-818a-cb4b198ee862-bundle\") pod \"0a36b25d-d9ad-4209-818a-cb4b198ee862\" (UID: \"0a36b25d-d9ad-4209-818a-cb4b198ee862\") " Nov 03 17:49:20.067974 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.067889 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f73fa28-8822-499b-907a-3bbfc864780c-kube-api-access-xbsvb\" (UniqueName: \"kubernetes.io/projected/8f73fa28-8822-499b-907a-3bbfc864780c-kube-api-access-xbsvb\") pod \"8f73fa28-8822-499b-907a-3bbfc864780c\" (UID: \"8f73fa28-8822-499b-907a-3bbfc864780c\") " Nov 03 17:49:20.067974 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.067915 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a36b25d-d9ad-4209-818a-cb4b198ee862-util\" (UniqueName: \"kubernetes.io/empty-dir/0a36b25d-d9ad-4209-818a-cb4b198ee862-util\") pod \"0a36b25d-d9ad-4209-818a-cb4b198ee862\" (UID: \"0a36b25d-d9ad-4209-818a-cb4b198ee862\") " Nov 03 17:49:20.067974 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.067945 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/951df12e-8840-405b-b461-a4fe2cf59485-util\" (UniqueName: \"kubernetes.io/empty-dir/951df12e-8840-405b-b461-a4fe2cf59485-util\") pod \"951df12e-8840-405b-b461-a4fe2cf59485\" (UID: \"951df12e-8840-405b-b461-a4fe2cf59485\") " Nov 03 17:49:20.067974 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.067955 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951df12e-8840-405b-b461-a4fe2cf59485-bundle" pod "951df12e-8840-405b-b461-a4fe2cf59485" (UID: "951df12e-8840-405b-b461-a4fe2cf59485"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:20.068300 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.068207 5805 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/951df12e-8840-405b-b461-a4fe2cf59485-bundle\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:20.069022 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.068570 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a36b25d-d9ad-4209-818a-cb4b198ee862-bundle" pod "0a36b25d-d9ad-4209-818a-cb4b198ee862" (UID: "0a36b25d-d9ad-4209-818a-cb4b198ee862"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:20.069712 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.069688 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f73fa28-8822-499b-907a-3bbfc864780c-bundle" pod "8f73fa28-8822-499b-907a-3bbfc864780c" (UID: "8f73fa28-8822-499b-907a-3bbfc864780c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:20.070005 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.069978 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a36b25d-d9ad-4209-818a-cb4b198ee862-kube-api-access-9hht6" pod "0a36b25d-d9ad-4209-818a-cb4b198ee862" (UID: "0a36b25d-d9ad-4209-818a-cb4b198ee862"). InnerVolumeSpecName "kube-api-access-9hht6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:49:20.073377 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.073312 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f73fa28-8822-499b-907a-3bbfc864780c-kube-api-access-xbsvb" pod "8f73fa28-8822-499b-907a-3bbfc864780c" (UID: "8f73fa28-8822-499b-907a-3bbfc864780c"). InnerVolumeSpecName "kube-api-access-xbsvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:49:20.073492 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.073439 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f73fa28-8822-499b-907a-3bbfc864780c-util" pod "8f73fa28-8822-499b-907a-3bbfc864780c" (UID: "8f73fa28-8822-499b-907a-3bbfc864780c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:20.073601 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.073578 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951df12e-8840-405b-b461-a4fe2cf59485-kube-api-access-cbpn6" pod "951df12e-8840-405b-b461-a4fe2cf59485" (UID: "951df12e-8840-405b-b461-a4fe2cf59485"). InnerVolumeSpecName "kube-api-access-cbpn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:49:20.074105 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.074085 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951df12e-8840-405b-b461-a4fe2cf59485-util" pod "951df12e-8840-405b-b461-a4fe2cf59485" (UID: "951df12e-8840-405b-b461-a4fe2cf59485"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:20.075674 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.075651 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a36b25d-d9ad-4209-818a-cb4b198ee862-util" pod "0a36b25d-d9ad-4209-818a-cb4b198ee862" (UID: "0a36b25d-d9ad-4209-818a-cb4b198ee862"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:20.169582 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.169548 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbsvb\" (UniqueName: \"kubernetes.io/projected/8f73fa28-8822-499b-907a-3bbfc864780c-kube-api-access-xbsvb\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:20.169582 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.169583 5805 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a36b25d-d9ad-4209-818a-cb4b198ee862-util\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:20.169789 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.169592 5805 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/951df12e-8840-405b-b461-a4fe2cf59485-util\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:20.169789 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.169601 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hht6\" (UniqueName: \"kubernetes.io/projected/0a36b25d-d9ad-4209-818a-cb4b198ee862-kube-api-access-9hht6\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:20.169789 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.169611 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbpn6\" (UniqueName: \"kubernetes.io/projected/951df12e-8840-405b-b461-a4fe2cf59485-kube-api-access-cbpn6\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:20.169789 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.169619 5805 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f73fa28-8822-499b-907a-3bbfc864780c-util\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:20.169789 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.169627 5805 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f73fa28-8822-499b-907a-3bbfc864780c-bundle\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:20.169789 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.169635 5805 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a36b25d-d9ad-4209-818a-cb4b198ee862-bundle\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:20.498223 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.498142 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:20.826002 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.825975 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf" Nov 03 17:49:20.826002 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.825993 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wzwgf" event={"ID":"0a36b25d-d9ad-4209-818a-cb4b198ee862","Type":"ContainerDied","Data":"016cdee04741a78fe4d202fc4ca02aed8e8ceade72843980757f8021259e8dad"} Nov 03 17:49:20.826504 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.826066 5805 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="016cdee04741a78fe4d202fc4ca02aed8e8ceade72843980757f8021259e8dad" Nov 03 17:49:20.828063 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.828038 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l" Nov 03 17:49:20.828194 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.828033 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75d45l" event={"ID":"951df12e-8840-405b-b461-a4fe2cf59485","Type":"ContainerDied","Data":"c5db01c03c8ff2fc5f3421e81a7ddd7200b3ef12091338423865c94196287d3f"} Nov 03 17:49:20.828194 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.828149 5805 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5db01c03c8ff2fc5f3421e81a7ddd7200b3ef12091338423865c94196287d3f" Nov 03 17:49:20.830216 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.830192 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87" event={"ID":"8f73fa28-8822-499b-907a-3bbfc864780c","Type":"ContainerDied","Data":"39c4213bdcb3955e6c5cf5585b672cdb8ce68950f5968e54926b39f4eb66963f"} Nov 03 17:49:20.830330 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.830226 5805 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39c4213bdcb3955e6c5cf5585b672cdb8ce68950f5968e54926b39f4eb66963f" Nov 03 17:49:20.830330 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:20.830239 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6sq87" Nov 03 17:49:22.840590 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:22.840558 5805 generic.go:356] "Generic (PLEG): container finished" podID="6dffb926-bfdc-47b8-aaab-b3cb58178091" containerID="0d7112bb94cf4fc88c4f36a2a7635aa76c775bbc824e963e41cf60f06a4c5b39" exitCode=0 Nov 03 17:49:22.840988 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:22.840647 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9" event={"ID":"6dffb926-bfdc-47b8-aaab-b3cb58178091","Type":"ContainerDied","Data":"0d7112bb94cf4fc88c4f36a2a7635aa76c775bbc824e963e41cf60f06a4c5b39"} Nov 03 17:49:23.846065 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:23.846030 5805 generic.go:356] "Generic (PLEG): container finished" podID="6dffb926-bfdc-47b8-aaab-b3cb58178091" containerID="3e7c6bb226856c2bd56bc83dc9ec63aef3e0ed4fdb957e08024303ddfc637188" exitCode=0 Nov 03 17:49:23.846452 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:23.846115 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9" event={"ID":"6dffb926-bfdc-47b8-aaab-b3cb58178091","Type":"ContainerDied","Data":"3e7c6bb226856c2bd56bc83dc9ec63aef3e0ed4fdb957e08024303ddfc637188"} Nov 03 17:49:24.889003 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:24.888973 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:24.968220 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:24.968197 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9" Nov 03 17:49:25.114616 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:25.114545 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dffb926-bfdc-47b8-aaab-b3cb58178091-kube-api-access-f9bgv\" (UniqueName: \"kubernetes.io/projected/6dffb926-bfdc-47b8-aaab-b3cb58178091-kube-api-access-f9bgv\") pod \"6dffb926-bfdc-47b8-aaab-b3cb58178091\" (UID: \"6dffb926-bfdc-47b8-aaab-b3cb58178091\") " Nov 03 17:49:25.114616 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:25.114614 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dffb926-bfdc-47b8-aaab-b3cb58178091-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6dffb926-bfdc-47b8-aaab-b3cb58178091-bundle\") pod \"6dffb926-bfdc-47b8-aaab-b3cb58178091\" (UID: \"6dffb926-bfdc-47b8-aaab-b3cb58178091\") " Nov 03 17:49:25.114842 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:25.114699 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dffb926-bfdc-47b8-aaab-b3cb58178091-util\" (UniqueName: \"kubernetes.io/empty-dir/6dffb926-bfdc-47b8-aaab-b3cb58178091-util\") pod \"6dffb926-bfdc-47b8-aaab-b3cb58178091\" (UID: \"6dffb926-bfdc-47b8-aaab-b3cb58178091\") " Nov 03 17:49:25.115070 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:25.115041 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dffb926-bfdc-47b8-aaab-b3cb58178091-bundle" pod "6dffb926-bfdc-47b8-aaab-b3cb58178091" (UID: "6dffb926-bfdc-47b8-aaab-b3cb58178091"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:25.116719 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:25.116688 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dffb926-bfdc-47b8-aaab-b3cb58178091-kube-api-access-f9bgv" pod "6dffb926-bfdc-47b8-aaab-b3cb58178091" (UID: "6dffb926-bfdc-47b8-aaab-b3cb58178091"). InnerVolumeSpecName "kube-api-access-f9bgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:49:25.119834 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:25.119805 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dffb926-bfdc-47b8-aaab-b3cb58178091-util" pod "6dffb926-bfdc-47b8-aaab-b3cb58178091" (UID: "6dffb926-bfdc-47b8-aaab-b3cb58178091"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:49:25.215935 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:25.215910 5805 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6dffb926-bfdc-47b8-aaab-b3cb58178091-util\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:25.215935 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:25.215935 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9bgv\" (UniqueName: \"kubernetes.io/projected/6dffb926-bfdc-47b8-aaab-b3cb58178091-kube-api-access-f9bgv\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:25.216091 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:25.215945 5805 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6dffb926-bfdc-47b8-aaab-b3cb58178091-bundle\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:49:25.855054 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:25.855014 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9" event={"ID":"6dffb926-bfdc-47b8-aaab-b3cb58178091","Type":"ContainerDied","Data":"5e8634705a070377c50747b57aaba2da12b56dcde78ee09f7968e90c198bdc63"} Nov 03 17:49:25.855194 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:25.855062 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kh9q9" Nov 03 17:49:25.855194 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:25.855068 5805 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e8634705a070377c50747b57aaba2da12b56dcde78ee09f7968e90c198bdc63" Nov 03 17:49:25.937045 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:25.937015 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:26.234258 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:49:26.234174 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:49:29.159308 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:29.159274 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:29.398237 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:29.398208 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:31.255537 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:31.255502 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:32.719318 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:32.719282 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:33.027074 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.026993 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:33.390408 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390363 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-46sr7"] Nov 03 17:49:33.390683 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390669 5805 cpu_manager.go:414] "containerMap: removing container" podUID="6dffb926-bfdc-47b8-aaab-b3cb58178091" containerName="util" Nov 03 17:49:33.390725 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390685 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dffb926-bfdc-47b8-aaab-b3cb58178091" containerName="util" Nov 03 17:49:33.390725 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390698 5805 cpu_manager.go:414] "containerMap: removing container" podUID="8f73fa28-8822-499b-907a-3bbfc864780c" containerName="util" Nov 03 17:49:33.390725 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390705 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f73fa28-8822-499b-907a-3bbfc864780c" containerName="util" Nov 03 17:49:33.390725 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390712 5805 cpu_manager.go:414] "containerMap: removing container" podUID="0a36b25d-d9ad-4209-818a-cb4b198ee862" containerName="pull" Nov 03 17:49:33.390725 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390717 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a36b25d-d9ad-4209-818a-cb4b198ee862" containerName="pull" Nov 03 17:49:33.390725 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390724 5805 cpu_manager.go:414] "containerMap: removing container" podUID="6dffb926-bfdc-47b8-aaab-b3cb58178091" containerName="extract" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390729 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dffb926-bfdc-47b8-aaab-b3cb58178091" containerName="extract" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390742 5805 cpu_manager.go:414] "containerMap: removing container" podUID="951df12e-8840-405b-b461-a4fe2cf59485" containerName="util" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390746 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="951df12e-8840-405b-b461-a4fe2cf59485" containerName="util" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390753 5805 cpu_manager.go:414] "containerMap: removing container" podUID="1ebebdbe-ef52-4178-8007-432690ec1dc0" containerName="pull" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390757 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ebebdbe-ef52-4178-8007-432690ec1dc0" containerName="pull" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390763 5805 cpu_manager.go:414] "containerMap: removing container" podUID="6dffb926-bfdc-47b8-aaab-b3cb58178091" containerName="pull" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390767 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dffb926-bfdc-47b8-aaab-b3cb58178091" containerName="pull" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390775 5805 cpu_manager.go:414] "containerMap: removing container" podUID="1ebebdbe-ef52-4178-8007-432690ec1dc0" containerName="extract" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390779 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ebebdbe-ef52-4178-8007-432690ec1dc0" containerName="extract" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390786 5805 cpu_manager.go:414] "containerMap: removing container" podUID="951df12e-8840-405b-b461-a4fe2cf59485" containerName="pull" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390791 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="951df12e-8840-405b-b461-a4fe2cf59485" containerName="pull" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390798 5805 cpu_manager.go:414] "containerMap: removing container" podUID="0a36b25d-d9ad-4209-818a-cb4b198ee862" containerName="extract" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390802 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a36b25d-d9ad-4209-818a-cb4b198ee862" containerName="extract" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390808 5805 cpu_manager.go:414] "containerMap: removing container" podUID="8f73fa28-8822-499b-907a-3bbfc864780c" containerName="extract" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390812 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f73fa28-8822-499b-907a-3bbfc864780c" containerName="extract" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390819 5805 cpu_manager.go:414] "containerMap: removing container" podUID="1ebebdbe-ef52-4178-8007-432690ec1dc0" containerName="util" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390823 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ebebdbe-ef52-4178-8007-432690ec1dc0" containerName="util" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390828 5805 cpu_manager.go:414] "containerMap: removing container" podUID="951df12e-8840-405b-b461-a4fe2cf59485" containerName="extract" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390833 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="951df12e-8840-405b-b461-a4fe2cf59485" containerName="extract" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390840 5805 cpu_manager.go:414] "containerMap: removing container" podUID="0a36b25d-d9ad-4209-818a-cb4b198ee862" containerName="util" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390844 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a36b25d-d9ad-4209-818a-cb4b198ee862" containerName="util" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390850 5805 cpu_manager.go:414] "containerMap: removing container" podUID="8f73fa28-8822-499b-907a-3bbfc864780c" containerName="pull" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390853 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f73fa28-8822-499b-907a-3bbfc864780c" containerName="pull" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390901 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dffb926-bfdc-47b8-aaab-b3cb58178091" containerName="extract" Nov 03 17:49:33.390896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390911 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ebebdbe-ef52-4178-8007-432690ec1dc0" containerName="extract" Nov 03 17:49:33.391635 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390917 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f73fa28-8822-499b-907a-3bbfc864780c" containerName="extract" Nov 03 17:49:33.391635 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390923 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a36b25d-d9ad-4209-818a-cb4b198ee862" containerName="extract" Nov 03 17:49:33.391635 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.390928 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="951df12e-8840-405b-b461-a4fe2cf59485" containerName="extract" Nov 03 17:49:33.393813 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.393792 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-46sr7" Nov 03 17:49:33.491973 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.491942 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 17:49:33.492206 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.492000 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6wwlb\"" Nov 03 17:49:33.492206 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.492173 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 17:49:33.493449 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.493432 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-46sr7"] Nov 03 17:49:33.494354 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.494337 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rh67\" (UniqueName: \"kubernetes.io/projected/01ee4c19-32e1-433a-88b6-d6f6209bae7f-kube-api-access-8rh67\") pod \"cert-manager-webhook-5dbbbf75cc-46sr7\" (UID: \"01ee4c19-32e1-433a-88b6-d6f6209bae7f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-46sr7" Nov 03 17:49:33.494434 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.494418 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01ee4c19-32e1-433a-88b6-d6f6209bae7f-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-46sr7\" (UID: \"01ee4c19-32e1-433a-88b6-d6f6209bae7f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-46sr7" Nov 03 17:49:33.595437 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.595378 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rh67\" (UniqueName: \"kubernetes.io/projected/01ee4c19-32e1-433a-88b6-d6f6209bae7f-kube-api-access-8rh67\") pod \"cert-manager-webhook-5dbbbf75cc-46sr7\" (UID: \"01ee4c19-32e1-433a-88b6-d6f6209bae7f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-46sr7" Nov 03 17:49:33.595667 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.595489 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01ee4c19-32e1-433a-88b6-d6f6209bae7f-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-46sr7\" (UID: \"01ee4c19-32e1-433a-88b6-d6f6209bae7f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-46sr7" Nov 03 17:49:33.670127 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.670062 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01ee4c19-32e1-433a-88b6-d6f6209bae7f-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-46sr7\" (UID: \"01ee4c19-32e1-433a-88b6-d6f6209bae7f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-46sr7" Nov 03 17:49:33.672447 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.672422 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rh67\" (UniqueName: \"kubernetes.io/projected/01ee4c19-32e1-433a-88b6-d6f6209bae7f-kube-api-access-8rh67\") pod \"cert-manager-webhook-5dbbbf75cc-46sr7\" (UID: \"01ee4c19-32e1-433a-88b6-d6f6209bae7f\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-46sr7" Nov 03 17:49:33.724425 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:33.724384 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-46sr7" Nov 03 17:49:34.113829 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:34.113790 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-46sr7"] Nov 03 17:49:34.120801 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:49:34.120774 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01ee4c19_32e1_433a_88b6_d6f6209bae7f.slice/crio-1e94af239fb54841ebe6711c37425c48ce8e5fc8889b2e780bb2f0c7b011ce94 WatchSource:0}: Error finding container 1e94af239fb54841ebe6711c37425c48ce8e5fc8889b2e780bb2f0c7b011ce94: Status 404 returned error can't find the container with id 1e94af239fb54841ebe6711c37425c48ce8e5fc8889b2e780bb2f0c7b011ce94 Nov 03 17:49:34.888730 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:34.888690 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-46sr7" event={"ID":"01ee4c19-32e1-433a-88b6-d6f6209bae7f","Type":"ContainerStarted","Data":"1e94af239fb54841ebe6711c37425c48ce8e5fc8889b2e780bb2f0c7b011ce94"} Nov 03 17:49:35.185751 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:35.185641 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:35.606471 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:35.606437 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:36.097407 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:36.097370 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:38.725809 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:38.725763 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:39.058765 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:39.058730 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:39.320002 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:39.319974 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-x77rf"] Nov 03 17:49:39.322512 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:39.322491 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-x77rf" Nov 03 17:49:39.379104 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:39.379076 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-x77rf"] Nov 03 17:49:39.423719 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:39.423689 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-v59wl\"" Nov 03 17:49:39.444294 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:39.444265 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7cf2\" (UniqueName: \"kubernetes.io/projected/051eb337-18d2-4c3b-9353-0c95c1766674-kube-api-access-s7cf2\") pod \"cert-manager-cainjector-5c57bbd4fc-x77rf\" (UID: \"051eb337-18d2-4c3b-9353-0c95c1766674\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-x77rf" Nov 03 17:49:39.444467 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:39.444309 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/051eb337-18d2-4c3b-9353-0c95c1766674-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-x77rf\" (UID: \"051eb337-18d2-4c3b-9353-0c95c1766674\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-x77rf" Nov 03 17:49:39.494845 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:39.494808 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:39.545084 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:39.545050 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7cf2\" (UniqueName: \"kubernetes.io/projected/051eb337-18d2-4c3b-9353-0c95c1766674-kube-api-access-s7cf2\") pod \"cert-manager-cainjector-5c57bbd4fc-x77rf\" (UID: \"051eb337-18d2-4c3b-9353-0c95c1766674\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-x77rf" Nov 03 17:49:39.545220 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:39.545098 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/051eb337-18d2-4c3b-9353-0c95c1766674-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-x77rf\" (UID: \"051eb337-18d2-4c3b-9353-0c95c1766674\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-x77rf" Nov 03 17:49:39.616433 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:39.616350 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/051eb337-18d2-4c3b-9353-0c95c1766674-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-x77rf\" (UID: \"051eb337-18d2-4c3b-9353-0c95c1766674\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-x77rf" Nov 03 17:49:39.617306 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:39.617285 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7cf2\" (UniqueName: \"kubernetes.io/projected/051eb337-18d2-4c3b-9353-0c95c1766674-kube-api-access-s7cf2\") pod \"cert-manager-cainjector-5c57bbd4fc-x77rf\" (UID: \"051eb337-18d2-4c3b-9353-0c95c1766674\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-x77rf" Nov 03 17:49:39.634145 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:39.634120 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-x77rf" Nov 03 17:49:39.913025 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:39.912941 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-46sr7" event={"ID":"01ee4c19-32e1-433a-88b6-d6f6209bae7f","Type":"ContainerStarted","Data":"acbddf9e389c532504e606972e23af0792583c800e3e51f668c61157b5534e26"} Nov 03 17:49:39.913459 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:39.913074 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-46sr7" Nov 03 17:49:40.011492 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.011422 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-x77rf"] Nov 03 17:49:40.012673 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.012648 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:40.018730 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:49:40.018698 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod051eb337_18d2_4c3b_9353_0c95c1766674.slice/crio-2d9c6ff02162fb64dbd4e9aa06642a7423b5db26029ec16532aa4d91d7ccdfe8 WatchSource:0}: Error finding container 2d9c6ff02162fb64dbd4e9aa06642a7423b5db26029ec16532aa4d91d7ccdfe8: Status 404 returned error can't find the container with id 2d9c6ff02162fb64dbd4e9aa06642a7423b5db26029ec16532aa4d91d7ccdfe8 Nov 03 17:49:40.032127 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.032067 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-46sr7" podStartSLOduration=2.21233839 podStartE2EDuration="7.032049505s" totalImagesPullingTime="4.819711115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:40.031535865 +0000 UTC m=+217.165164897" watchObservedRunningTime="2026-11-03 17:49:40.032049505 +0000 UTC m=+217.165678517" Nov 03 17:49:40.188240 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.188163 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk"] Nov 03 17:49:40.190970 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.190956 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" Nov 03 17:49:40.261554 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.261501 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 17:49:40.262336 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.262200 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jdbrd\"" Nov 03 17:49:40.262336 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.262269 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 17:49:40.262336 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.262327 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 17:49:40.269329 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.269314 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 17:49:40.269420 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.269342 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 17:49:40.309106 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.309066 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk"] Nov 03 17:49:40.347129 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.347095 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:40.352305 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.352279 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8cea489-7c16-4fb9-92ff-8c52371e032d-tmp\") pod \"opendatahub-operator-controller-manager-6dd86677fd-phkrk\" (UID: \"d8cea489-7c16-4fb9-92ff-8c52371e032d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" Nov 03 17:49:40.352480 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.352312 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d8cea489-7c16-4fb9-92ff-8c52371e032d-manifests\") pod \"opendatahub-operator-controller-manager-6dd86677fd-phkrk\" (UID: \"d8cea489-7c16-4fb9-92ff-8c52371e032d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" Nov 03 17:49:40.352480 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.352339 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d8cea489-7c16-4fb9-92ff-8c52371e032d-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-phkrk\" (UID: \"d8cea489-7c16-4fb9-92ff-8c52371e032d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" Nov 03 17:49:40.352480 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.352424 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d8cea489-7c16-4fb9-92ff-8c52371e032d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-phkrk\" (UID: \"d8cea489-7c16-4fb9-92ff-8c52371e032d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" Nov 03 17:49:40.352606 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.352529 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prr82\" (UniqueName: \"kubernetes.io/projected/d8cea489-7c16-4fb9-92ff-8c52371e032d-kube-api-access-prr82\") pod \"opendatahub-operator-controller-manager-6dd86677fd-phkrk\" (UID: \"d8cea489-7c16-4fb9-92ff-8c52371e032d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" Nov 03 17:49:40.352606 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.352566 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8cea489-7c16-4fb9-92ff-8c52371e032d-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-phkrk\" (UID: \"d8cea489-7c16-4fb9-92ff-8c52371e032d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" Nov 03 17:49:40.453657 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.453571 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prr82\" (UniqueName: \"kubernetes.io/projected/d8cea489-7c16-4fb9-92ff-8c52371e032d-kube-api-access-prr82\") pod \"opendatahub-operator-controller-manager-6dd86677fd-phkrk\" (UID: \"d8cea489-7c16-4fb9-92ff-8c52371e032d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" Nov 03 17:49:40.453657 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.453622 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8cea489-7c16-4fb9-92ff-8c52371e032d-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-phkrk\" (UID: \"d8cea489-7c16-4fb9-92ff-8c52371e032d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" Nov 03 17:49:40.453657 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.453658 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8cea489-7c16-4fb9-92ff-8c52371e032d-tmp\") pod \"opendatahub-operator-controller-manager-6dd86677fd-phkrk\" (UID: \"d8cea489-7c16-4fb9-92ff-8c52371e032d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" Nov 03 17:49:40.453894 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.453686 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d8cea489-7c16-4fb9-92ff-8c52371e032d-manifests\") pod \"opendatahub-operator-controller-manager-6dd86677fd-phkrk\" (UID: \"d8cea489-7c16-4fb9-92ff-8c52371e032d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" Nov 03 17:49:40.453894 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.453800 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d8cea489-7c16-4fb9-92ff-8c52371e032d-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-phkrk\" (UID: \"d8cea489-7c16-4fb9-92ff-8c52371e032d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" Nov 03 17:49:40.453894 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.453846 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d8cea489-7c16-4fb9-92ff-8c52371e032d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-phkrk\" (UID: \"d8cea489-7c16-4fb9-92ff-8c52371e032d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" Nov 03 17:49:40.454308 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.454264 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d8cea489-7c16-4fb9-92ff-8c52371e032d-manifests\") pod \"opendatahub-operator-controller-manager-6dd86677fd-phkrk\" (UID: \"d8cea489-7c16-4fb9-92ff-8c52371e032d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" Nov 03 17:49:40.455980 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.455953 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8cea489-7c16-4fb9-92ff-8c52371e032d-tmp\") pod \"opendatahub-operator-controller-manager-6dd86677fd-phkrk\" (UID: \"d8cea489-7c16-4fb9-92ff-8c52371e032d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" Nov 03 17:49:40.456270 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.456238 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d8cea489-7c16-4fb9-92ff-8c52371e032d-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-phkrk\" (UID: \"d8cea489-7c16-4fb9-92ff-8c52371e032d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" Nov 03 17:49:40.456694 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.456676 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d8cea489-7c16-4fb9-92ff-8c52371e032d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-phkrk\" (UID: \"d8cea489-7c16-4fb9-92ff-8c52371e032d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" Nov 03 17:49:40.456769 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.456690 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8cea489-7c16-4fb9-92ff-8c52371e032d-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dd86677fd-phkrk\" (UID: \"d8cea489-7c16-4fb9-92ff-8c52371e032d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" Nov 03 17:49:40.558309 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.558265 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prr82\" (UniqueName: \"kubernetes.io/projected/d8cea489-7c16-4fb9-92ff-8c52371e032d-kube-api-access-prr82\") pod \"opendatahub-operator-controller-manager-6dd86677fd-phkrk\" (UID: \"d8cea489-7c16-4fb9-92ff-8c52371e032d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" Nov 03 17:49:40.729651 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.729551 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:40.804841 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.804808 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" Nov 03 17:49:40.920066 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.920024 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-x77rf" event={"ID":"051eb337-18d2-4c3b-9353-0c95c1766674","Type":"ContainerStarted","Data":"6c2b58a0b8f4ccf962721fd4d3eff0f59af1d8a546a289d737d0da7e71c5a102"} Nov 03 17:49:40.920513 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:40.920076 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-x77rf" event={"ID":"051eb337-18d2-4c3b-9353-0c95c1766674","Type":"ContainerStarted","Data":"2d9c6ff02162fb64dbd4e9aa06642a7423b5db26029ec16532aa4d91d7ccdfe8"} Nov 03 17:49:41.055250 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:41.055167 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:41.068274 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:41.068218 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-x77rf" podStartSLOduration=2.068204329 podStartE2EDuration="2.068204329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:41.067816218 +0000 UTC m=+218.201445256" watchObservedRunningTime="2026-11-03 17:49:41.068204329 +0000 UTC m=+218.201833333" Nov 03 17:49:41.431550 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:41.430560 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk"] Nov 03 17:49:41.438115 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:49:41.438083 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8cea489_7c16_4fb9_92ff_8c52371e032d.slice/crio-559d9c38b3258d60c5d9684499ce3ca1b46e08fa19fdaeee298718471aacd072 WatchSource:0}: Error finding container 559d9c38b3258d60c5d9684499ce3ca1b46e08fa19fdaeee298718471aacd072: Status 404 returned error can't find the container with id 559d9c38b3258d60c5d9684499ce3ca1b46e08fa19fdaeee298718471aacd072 Nov 03 17:49:41.480992 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:41.480881 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:41.925903 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:41.925861 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" event={"ID":"d8cea489-7c16-4fb9-92ff-8c52371e032d","Type":"ContainerStarted","Data":"559d9c38b3258d60c5d9684499ce3ca1b46e08fa19fdaeee298718471aacd072"} Nov 03 17:49:42.372387 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:42.372355 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:42.645278 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:42.645194 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:43.334538 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:43.334503 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:43.444907 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:43.444868 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-spd46"] Nov 03 17:49:43.448519 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:43.448493 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-spd46" Nov 03 17:49:43.477247 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:43.477219 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/142d6678-a7b6-49c3-8061-1bc80231614d-bound-sa-token\") pod \"cert-manager-597b95cb84-spd46\" (UID: \"142d6678-a7b6-49c3-8061-1bc80231614d\") " pod="cert-manager/cert-manager-597b95cb84-spd46" Nov 03 17:49:43.477388 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:43.477275 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4d5z\" (UniqueName: \"kubernetes.io/projected/142d6678-a7b6-49c3-8061-1bc80231614d-kube-api-access-v4d5z\") pod \"cert-manager-597b95cb84-spd46\" (UID: \"142d6678-a7b6-49c3-8061-1bc80231614d\") " pod="cert-manager/cert-manager-597b95cb84-spd46" Nov 03 17:49:43.544198 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:43.544166 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-j8kvm\"" Nov 03 17:49:43.546310 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:43.546272 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-spd46"] Nov 03 17:49:43.577729 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:43.577698 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/142d6678-a7b6-49c3-8061-1bc80231614d-bound-sa-token\") pod \"cert-manager-597b95cb84-spd46\" (UID: \"142d6678-a7b6-49c3-8061-1bc80231614d\") " pod="cert-manager/cert-manager-597b95cb84-spd46" Nov 03 17:49:43.577855 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:43.577758 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4d5z\" (UniqueName: \"kubernetes.io/projected/142d6678-a7b6-49c3-8061-1bc80231614d-kube-api-access-v4d5z\") pod \"cert-manager-597b95cb84-spd46\" (UID: \"142d6678-a7b6-49c3-8061-1bc80231614d\") " pod="cert-manager/cert-manager-597b95cb84-spd46" Nov 03 17:49:43.642586 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:43.642521 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4d5z\" (UniqueName: \"kubernetes.io/projected/142d6678-a7b6-49c3-8061-1bc80231614d-kube-api-access-v4d5z\") pod \"cert-manager-597b95cb84-spd46\" (UID: \"142d6678-a7b6-49c3-8061-1bc80231614d\") " pod="cert-manager/cert-manager-597b95cb84-spd46" Nov 03 17:49:43.642713 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:43.642587 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/142d6678-a7b6-49c3-8061-1bc80231614d-bound-sa-token\") pod \"cert-manager-597b95cb84-spd46\" (UID: \"142d6678-a7b6-49c3-8061-1bc80231614d\") " pod="cert-manager/cert-manager-597b95cb84-spd46" Nov 03 17:49:43.761428 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:43.761388 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-spd46" Nov 03 17:49:44.228242 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:44.228208 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-spd46"] Nov 03 17:49:44.235023 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:49:44.234991 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod142d6678_a7b6_49c3_8061_1bc80231614d.slice/crio-c96eaf545f32e8dc728e866c830ae48130294cb10726824a3c06e88b4c94e699 WatchSource:0}: Error finding container c96eaf545f32e8dc728e866c830ae48130294cb10726824a3c06e88b4c94e699: Status 404 returned error can't find the container with id c96eaf545f32e8dc728e866c830ae48130294cb10726824a3c06e88b4c94e699 Nov 03 17:49:44.390767 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:44.390724 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-pv7ng"] Nov 03 17:49:44.394341 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:44.394310 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pv7ng" Nov 03 17:49:44.461173 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:44.461142 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-z9gc9\"" Nov 03 17:49:44.461173 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:44.461142 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 17:49:44.461429 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:44.461258 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 17:49:44.467041 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:44.467007 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-pv7ng"] Nov 03 17:49:44.485974 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:44.485905 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsxkg\" (UniqueName: \"kubernetes.io/projected/bdfb81c0-54f8-425e-8b8d-703158337ed4-kube-api-access-tsxkg\") pod \"jobset-operator-674cd59dc5-pv7ng\" (UID: \"bdfb81c0-54f8-425e-8b8d-703158337ed4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pv7ng" Nov 03 17:49:44.486092 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:44.486046 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdfb81c0-54f8-425e-8b8d-703158337ed4-tmp\") pod \"jobset-operator-674cd59dc5-pv7ng\" (UID: \"bdfb81c0-54f8-425e-8b8d-703158337ed4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pv7ng" Nov 03 17:49:44.586829 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:44.586794 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsxkg\" (UniqueName: \"kubernetes.io/projected/bdfb81c0-54f8-425e-8b8d-703158337ed4-kube-api-access-tsxkg\") pod \"jobset-operator-674cd59dc5-pv7ng\" (UID: \"bdfb81c0-54f8-425e-8b8d-703158337ed4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pv7ng" Nov 03 17:49:44.587029 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:44.586883 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdfb81c0-54f8-425e-8b8d-703158337ed4-tmp\") pod \"jobset-operator-674cd59dc5-pv7ng\" (UID: \"bdfb81c0-54f8-425e-8b8d-703158337ed4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pv7ng" Nov 03 17:49:44.587306 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:44.587283 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bdfb81c0-54f8-425e-8b8d-703158337ed4-tmp\") pod \"jobset-operator-674cd59dc5-pv7ng\" (UID: \"bdfb81c0-54f8-425e-8b8d-703158337ed4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pv7ng" Nov 03 17:49:44.651836 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:44.651805 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsxkg\" (UniqueName: \"kubernetes.io/projected/bdfb81c0-54f8-425e-8b8d-703158337ed4-kube-api-access-tsxkg\") pod \"jobset-operator-674cd59dc5-pv7ng\" (UID: \"bdfb81c0-54f8-425e-8b8d-703158337ed4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pv7ng" Nov 03 17:49:44.708131 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:44.708099 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pv7ng" Nov 03 17:49:44.827830 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:44.827805 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:44.943459 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:44.943425 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-spd46" event={"ID":"142d6678-a7b6-49c3-8061-1bc80231614d","Type":"ContainerStarted","Data":"f24059e9ca8f37137710ea7202b4310700c48bcaf94e163dc9e5f13ac315acf4"} Nov 03 17:49:44.943687 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:44.943468 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-spd46" event={"ID":"142d6678-a7b6-49c3-8061-1bc80231614d","Type":"ContainerStarted","Data":"c96eaf545f32e8dc728e866c830ae48130294cb10726824a3c06e88b4c94e699"} Nov 03 17:49:45.061674 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:45.061369 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-spd46" podStartSLOduration=2.061351078 podStartE2EDuration="2.061351078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:45.060747519 +0000 UTC m=+222.194376526" watchObservedRunningTime="2026-11-03 17:49:45.061351078 +0000 UTC m=+222.194980089" Nov 03 17:49:45.106016 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:45.105983 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-pv7ng"] Nov 03 17:49:45.121170 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:49:45.121143 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdfb81c0_54f8_425e_8b8d_703158337ed4.slice/crio-04e024fed8bec3fc4d4246f99b22183388c5c03706db460cda04358ab25a83df WatchSource:0}: Error finding container 04e024fed8bec3fc4d4246f99b22183388c5c03706db460cda04358ab25a83df: Status 404 returned error can't find the container with id 04e024fed8bec3fc4d4246f99b22183388c5c03706db460cda04358ab25a83df Nov 03 17:49:45.186570 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:45.186540 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:45.871459 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:45.871428 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:45.922173 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:45.922138 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-46sr7" Nov 03 17:49:45.950042 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:45.950006 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pv7ng" event={"ID":"bdfb81c0-54f8-425e-8b8d-703158337ed4","Type":"ContainerStarted","Data":"04e024fed8bec3fc4d4246f99b22183388c5c03706db460cda04358ab25a83df"} Nov 03 17:49:46.650371 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:46.650340 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:46.956991 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:46.956960 5805 generic.go:356] "Generic (PLEG): container finished" podID="d8cea489-7c16-4fb9-92ff-8c52371e032d" containerID="3bf3a099149838be537e12ebed4542fc17884ecab060f8bf296cb1b17a52b8eb" exitCode=0 Nov 03 17:49:46.957449 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:46.957039 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" event={"ID":"d8cea489-7c16-4fb9-92ff-8c52371e032d","Type":"ContainerDied","Data":"3bf3a099149838be537e12ebed4542fc17884ecab060f8bf296cb1b17a52b8eb"} Nov 03 17:49:47.547037 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:47.547006 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:48.967613 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:48.967571 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pv7ng" event={"ID":"bdfb81c0-54f8-425e-8b8d-703158337ed4","Type":"ContainerStarted","Data":"a7ec646f62b9f45f2f9a8843a4f7dde7b31ca1f5257557529aef5e99ab17dc35"} Nov 03 17:49:48.973257 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:48.973226 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" event={"ID":"d8cea489-7c16-4fb9-92ff-8c52371e032d","Type":"ContainerStarted","Data":"8bc37c91390b3b73245af5c131c6f17e1ba95ff2271b32400c73d06b56e01222"} Nov 03 17:49:48.973587 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:48.973286 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" Nov 03 17:49:49.083563 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:49.083506 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-pv7ng" podStartSLOduration=1.448085128 podStartE2EDuration="5.083488331s" totalImagesPullingTime="3.635403203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:49.082467614 +0000 UTC m=+226.216096644" watchObservedRunningTime="2026-11-03 17:49:49.083488331 +0000 UTC m=+226.217117342" Nov 03 17:49:49.374849 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:49.374755 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" podStartSLOduration=2.958780708 podStartE2EDuration="9.374740365s" totalImagesPullingTime="6.013407386s" totalInitContainerRuntime="402.552271ms" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:49.327880575 +0000 UTC m=+226.461509605" watchObservedRunningTime="2026-11-03 17:49:49.374740365 +0000 UTC m=+226.508369375" Nov 03 17:49:50.130318 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:50.130279 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:52.142688 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:52.142658 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:52.379452 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:52.379414 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:54.939170 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:54.939136 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:55.193647 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:55.193559 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:56.234133 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:49:56.234099 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:49:58.867755 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:58.867715 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:59.100896 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:59.100861 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:49:59.976788 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:49:59.976756 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd86677fd-phkrk" Nov 03 17:50:04.348944 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:04.348910 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:04.656353 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:04.656268 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:05.240776 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:05.240744 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-4k7pg"] Nov 03 17:50:05.247885 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:05.247864 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4k7pg" Nov 03 17:50:05.272322 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:05.272291 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0c040730-05c1-4271-8c50-7ba7424d3205-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-4k7pg\" (UID: \"0c040730-05c1-4271-8c50-7ba7424d3205\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4k7pg" Nov 03 17:50:05.272468 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:05.272425 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c040730-05c1-4271-8c50-7ba7424d3205-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-4k7pg\" (UID: \"0c040730-05c1-4271-8c50-7ba7424d3205\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4k7pg" Nov 03 17:50:05.272565 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:05.272547 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdjhx\" (UniqueName: \"kubernetes.io/projected/0c040730-05c1-4271-8c50-7ba7424d3205-kube-api-access-xdjhx\") pod \"kuadrant-console-plugin-66c4fb575d-4k7pg\" (UID: \"0c040730-05c1-4271-8c50-7ba7424d3205\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4k7pg" Nov 03 17:50:05.318724 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:05.318695 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hj9sf\"" Nov 03 17:50:05.318869 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:05.318758 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 03 17:50:05.353802 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:05.353765 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 17:50:05.353802 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:05.353786 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 17:50:05.356120 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:05.356099 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 03 17:50:05.358374 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:05.358348 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-4k7pg"] Nov 03 17:50:05.373550 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:05.373525 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c040730-05c1-4271-8c50-7ba7424d3205-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-4k7pg\" (UID: \"0c040730-05c1-4271-8c50-7ba7424d3205\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4k7pg" Nov 03 17:50:05.373673 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:05.373601 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdjhx\" (UniqueName: \"kubernetes.io/projected/0c040730-05c1-4271-8c50-7ba7424d3205-kube-api-access-xdjhx\") pod \"kuadrant-console-plugin-66c4fb575d-4k7pg\" (UID: \"0c040730-05c1-4271-8c50-7ba7424d3205\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4k7pg" Nov 03 17:50:05.373777 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:05.373758 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0c040730-05c1-4271-8c50-7ba7424d3205-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-4k7pg\" (UID: \"0c040730-05c1-4271-8c50-7ba7424d3205\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4k7pg" Nov 03 17:50:05.374605 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:05.374583 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0c040730-05c1-4271-8c50-7ba7424d3205-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-4k7pg\" (UID: \"0c040730-05c1-4271-8c50-7ba7424d3205\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4k7pg" Nov 03 17:50:05.376051 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:05.376026 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0c040730-05c1-4271-8c50-7ba7424d3205-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-4k7pg\" (UID: \"0c040730-05c1-4271-8c50-7ba7424d3205\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4k7pg" Nov 03 17:50:05.415268 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:05.415232 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:05.475106 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:05.475077 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdjhx\" (UniqueName: \"kubernetes.io/projected/0c040730-05c1-4271-8c50-7ba7424d3205-kube-api-access-xdjhx\") pod \"kuadrant-console-plugin-66c4fb575d-4k7pg\" (UID: \"0c040730-05c1-4271-8c50-7ba7424d3205\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4k7pg" Nov 03 17:50:05.560068 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:05.560042 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4k7pg" Nov 03 17:50:05.702566 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:05.702532 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:05.937465 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:05.936683 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-4k7pg"] Nov 03 17:50:05.945856 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:50:05.945812 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c040730_05c1_4271_8c50_7ba7424d3205.slice/crio-8876e6ffa7e18ceb83589cbc4dde675c74d734dd15e7f43b341c5c4882b871c9 WatchSource:0}: Error finding container 8876e6ffa7e18ceb83589cbc4dde675c74d734dd15e7f43b341c5c4882b871c9: Status 404 returned error can't find the container with id 8876e6ffa7e18ceb83589cbc4dde675c74d734dd15e7f43b341c5c4882b871c9 Nov 03 17:50:06.038088 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:06.038054 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4k7pg" event={"ID":"0c040730-05c1-4271-8c50-7ba7424d3205","Type":"ContainerStarted","Data":"8876e6ffa7e18ceb83589cbc4dde675c74d734dd15e7f43b341c5c4882b871c9"} Nov 03 17:50:06.263018 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:06.262928 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:06.562837 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:06.562798 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:08.748672 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:08.748634 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:09.093875 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:09.093834 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:13.072803 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:13.072766 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4k7pg" event={"ID":"0c040730-05c1-4271-8c50-7ba7424d3205","Type":"ContainerStarted","Data":"b2bb0df50cd8fcf3d79c38957bddfff907f877d015d9f67942b056fdb4dd3814"} Nov 03 17:50:13.199913 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:13.199863 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4k7pg" podStartSLOduration=2.078196789 podStartE2EDuration="8.199849488s" totalImagesPullingTime="6.121652699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:50:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:50:13.198427613 +0000 UTC m=+250.332056623" watchObservedRunningTime="2026-11-03 17:50:13.199849488 +0000 UTC m=+250.333478498" Nov 03 17:50:15.155791 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:15.155754 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:15.368554 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:15.368521 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:17.057852 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:17.057821 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:17.367141 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:17.367050 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:17.747709 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:17.747636 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:18.184837 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:18.184799 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:18.516115 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:18.516038 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:19.168213 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:19.168180 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54df9ccd9f-v8bvm"] Nov 03 17:50:19.176267 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:19.176237 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-v8bvm" Nov 03 17:50:19.201721 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:19.201691 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:19.275341 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:19.275312 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-nrzqr\"" Nov 03 17:50:19.275341 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:19.275308 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 17:50:19.275341 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:19.275357 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 17:50:19.276694 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:19.276667 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54df9ccd9f-v8bvm"] Nov 03 17:50:19.302427 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:19.302380 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5aa44e3-5d93-4f17-869f-aa09922776c0-tmp\") pod \"kube-auth-proxy-54df9ccd9f-v8bvm\" (UID: \"a5aa44e3-5d93-4f17-869f-aa09922776c0\") " pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-v8bvm" Nov 03 17:50:19.302553 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:19.302448 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5aa44e3-5d93-4f17-869f-aa09922776c0-tls-certs\") pod \"kube-auth-proxy-54df9ccd9f-v8bvm\" (UID: \"a5aa44e3-5d93-4f17-869f-aa09922776c0\") " pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-v8bvm" Nov 03 17:50:19.302611 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:19.302570 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4f5b\" (UniqueName: \"kubernetes.io/projected/a5aa44e3-5d93-4f17-869f-aa09922776c0-kube-api-access-n4f5b\") pod \"kube-auth-proxy-54df9ccd9f-v8bvm\" (UID: \"a5aa44e3-5d93-4f17-869f-aa09922776c0\") " pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-v8bvm" Nov 03 17:50:19.403272 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:19.403237 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5aa44e3-5d93-4f17-869f-aa09922776c0-tmp\") pod \"kube-auth-proxy-54df9ccd9f-v8bvm\" (UID: \"a5aa44e3-5d93-4f17-869f-aa09922776c0\") " pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-v8bvm" Nov 03 17:50:19.403272 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:19.403276 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5aa44e3-5d93-4f17-869f-aa09922776c0-tls-certs\") pod \"kube-auth-proxy-54df9ccd9f-v8bvm\" (UID: \"a5aa44e3-5d93-4f17-869f-aa09922776c0\") " pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-v8bvm" Nov 03 17:50:19.403510 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:19.403427 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4f5b\" (UniqueName: \"kubernetes.io/projected/a5aa44e3-5d93-4f17-869f-aa09922776c0-kube-api-access-n4f5b\") pod \"kube-auth-proxy-54df9ccd9f-v8bvm\" (UID: \"a5aa44e3-5d93-4f17-869f-aa09922776c0\") " pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-v8bvm" Nov 03 17:50:19.405502 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:19.405479 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5aa44e3-5d93-4f17-869f-aa09922776c0-tmp\") pod \"kube-auth-proxy-54df9ccd9f-v8bvm\" (UID: \"a5aa44e3-5d93-4f17-869f-aa09922776c0\") " pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-v8bvm" Nov 03 17:50:19.405719 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:19.405701 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5aa44e3-5d93-4f17-869f-aa09922776c0-tls-certs\") pod \"kube-auth-proxy-54df9ccd9f-v8bvm\" (UID: \"a5aa44e3-5d93-4f17-869f-aa09922776c0\") " pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-v8bvm" Nov 03 17:50:19.458135 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:19.458041 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:19.461295 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:19.461269 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4f5b\" (UniqueName: \"kubernetes.io/projected/a5aa44e3-5d93-4f17-869f-aa09922776c0-kube-api-access-n4f5b\") pod \"kube-auth-proxy-54df9ccd9f-v8bvm\" (UID: \"a5aa44e3-5d93-4f17-869f-aa09922776c0\") " pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-v8bvm" Nov 03 17:50:19.487480 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:19.487447 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-v8bvm" Nov 03 17:50:19.753465 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:19.753355 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:19.894833 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:19.894804 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54df9ccd9f-v8bvm"] Nov 03 17:50:19.902102 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:50:19.902070 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5aa44e3_5d93_4f17_869f_aa09922776c0.slice/crio-d6dcef7eff978d52c876bd2137b64996dc6d345995b855d7076407f21b468bf6 WatchSource:0}: Error finding container d6dcef7eff978d52c876bd2137b64996dc6d345995b855d7076407f21b468bf6: Status 404 returned error can't find the container with id d6dcef7eff978d52c876bd2137b64996dc6d345995b855d7076407f21b468bf6 Nov 03 17:50:20.024005 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:20.023923 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:20.106647 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:20.106616 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-v8bvm" event={"ID":"a5aa44e3-5d93-4f17-869f-aa09922776c0","Type":"ContainerStarted","Data":"d6dcef7eff978d52c876bd2137b64996dc6d345995b855d7076407f21b468bf6"} Nov 03 17:50:20.392276 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:20.392244 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:20.722773 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:20.722697 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9g5jz"] Nov 03 17:50:20.726681 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:20.726657 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9g5jz" Nov 03 17:50:20.727776 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:20.727751 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:20.777153 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:20.777125 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 17:50:20.815304 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:20.815268 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbglj\" (UniqueName: \"kubernetes.io/projected/db4c7c05-8a9f-4134-a0d7-69ee24d4c787-kube-api-access-pbglj\") pod \"limitador-limitador-74dfcfb894-9g5jz\" (UID: \"db4c7c05-8a9f-4134-a0d7-69ee24d4c787\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9g5jz" Nov 03 17:50:20.815463 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:20.815309 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/db4c7c05-8a9f-4134-a0d7-69ee24d4c787-config-file\") pod \"limitador-limitador-74dfcfb894-9g5jz\" (UID: \"db4c7c05-8a9f-4134-a0d7-69ee24d4c787\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9g5jz" Nov 03 17:50:20.823114 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:20.823089 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9g5jz"] Nov 03 17:50:20.916283 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:20.916253 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbglj\" (UniqueName: \"kubernetes.io/projected/db4c7c05-8a9f-4134-a0d7-69ee24d4c787-kube-api-access-pbglj\") pod \"limitador-limitador-74dfcfb894-9g5jz\" (UID: \"db4c7c05-8a9f-4134-a0d7-69ee24d4c787\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9g5jz" Nov 03 17:50:20.916283 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:20.916289 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/db4c7c05-8a9f-4134-a0d7-69ee24d4c787-config-file\") pod \"limitador-limitador-74dfcfb894-9g5jz\" (UID: \"db4c7c05-8a9f-4134-a0d7-69ee24d4c787\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9g5jz" Nov 03 17:50:20.917016 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:20.916997 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/db4c7c05-8a9f-4134-a0d7-69ee24d4c787-config-file\") pod \"limitador-limitador-74dfcfb894-9g5jz\" (UID: \"db4c7c05-8a9f-4134-a0d7-69ee24d4c787\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9g5jz" Nov 03 17:50:20.984662 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:20.984576 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9g5jz"] Nov 03 17:50:20.996601 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:20.996570 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbglj\" (UniqueName: \"kubernetes.io/projected/db4c7c05-8a9f-4134-a0d7-69ee24d4c787-kube-api-access-pbglj\") pod \"limitador-limitador-74dfcfb894-9g5jz\" (UID: \"db4c7c05-8a9f-4134-a0d7-69ee24d4c787\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9g5jz" Nov 03 17:50:21.038493 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:21.038460 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9g5jz" Nov 03 17:50:21.075502 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:21.075442 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:21.311856 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:21.311821 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:21.425130 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:21.425093 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9g5jz"] Nov 03 17:50:21.439548 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:50:21.439519 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4c7c05_8a9f_4134_a0d7_69ee24d4c787.slice/crio-98f4c40a7c35866e515ea51518d3ed7b08b0b98e3fbc1ae8f18e634f6c73d952 WatchSource:0}: Error finding container 98f4c40a7c35866e515ea51518d3ed7b08b0b98e3fbc1ae8f18e634f6c73d952: Status 404 returned error can't find the container with id 98f4c40a7c35866e515ea51518d3ed7b08b0b98e3fbc1ae8f18e634f6c73d952 Nov 03 17:50:22.121383 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:22.121343 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9g5jz" event={"ID":"db4c7c05-8a9f-4134-a0d7-69ee24d4c787","Type":"ContainerStarted","Data":"98f4c40a7c35866e515ea51518d3ed7b08b0b98e3fbc1ae8f18e634f6c73d952"} Nov 03 17:50:22.465036 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:22.464961 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:22.709861 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:22.709822 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:23.093209 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:23.093179 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:23.127523 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:23.127486 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-v8bvm" event={"ID":"a5aa44e3-5d93-4f17-869f-aa09922776c0","Type":"ContainerStarted","Data":"d1788f617111b3be7a4b8c35993619e9a5bfbfd1c8270459c34f217cc831189b"} Nov 03 17:50:23.258558 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:23.257763 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54df9ccd9f-v8bvm" podStartSLOduration=1.355628543 podStartE2EDuration="4.257741688s" totalImagesPullingTime="2.902113145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:50:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:50:23.254519164 +0000 UTC m=+260.388148198" watchObservedRunningTime="2026-11-03 17:50:23.257741688 +0000 UTC m=+260.391370699" Nov 03 17:50:23.479084 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:23.479007 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:25.168764 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:25.168695 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:26.149566 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:26.149533 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9g5jz" event={"ID":"db4c7c05-8a9f-4134-a0d7-69ee24d4c787","Type":"ContainerStarted","Data":"1dd3779ff03049325e2444ea977b6a62bd07dcdb210630f19b915831f1696e93"} Nov 03 17:50:26.149734 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:26.149604 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-9g5jz" Nov 03 17:50:26.233631 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:50:26.233597 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:50:26.261910 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:26.261847 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-9g5jz" podStartSLOduration=2.170950346 podStartE2EDuration="6.261828162s" totalImagesPullingTime="4.090877816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:50:26.259965472 +0000 UTC m=+263.393594508" watchObservedRunningTime="2026-11-03 17:50:26.261828162 +0000 UTC m=+263.395457173" Nov 03 17:50:27.489057 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:27.489023 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:28.383342 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:28.383311 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:28.774246 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:28.774168 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:29.013749 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:29.013719 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:37.155248 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:37.155218 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-9g5jz" Nov 03 17:50:40.550113 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:40.550073 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9g5jz"] Nov 03 17:50:40.550650 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:40.550322 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-9g5jz" podUID="db4c7c05-8a9f-4134-a0d7-69ee24d4c787" containerName="limitador" containerID="cri-o://1dd3779ff03049325e2444ea977b6a62bd07dcdb210630f19b915831f1696e93" gracePeriod=30 Nov 03 17:50:40.584606 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:40.584574 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:40.790926 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:40.790887 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:41.092457 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:41.092435 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9g5jz" Nov 03 17:50:41.198913 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:41.198831 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db4c7c05-8a9f-4134-a0d7-69ee24d4c787-config-file\" (UniqueName: \"kubernetes.io/configmap/db4c7c05-8a9f-4134-a0d7-69ee24d4c787-config-file\") pod \"db4c7c05-8a9f-4134-a0d7-69ee24d4c787\" (UID: \"db4c7c05-8a9f-4134-a0d7-69ee24d4c787\") " Nov 03 17:50:41.198913 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:41.198900 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4c7c05-8a9f-4134-a0d7-69ee24d4c787-kube-api-access-pbglj\" (UniqueName: \"kubernetes.io/projected/db4c7c05-8a9f-4134-a0d7-69ee24d4c787-kube-api-access-pbglj\") pod \"db4c7c05-8a9f-4134-a0d7-69ee24d4c787\" (UID: \"db4c7c05-8a9f-4134-a0d7-69ee24d4c787\") " Nov 03 17:50:41.199235 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:41.199215 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db4c7c05-8a9f-4134-a0d7-69ee24d4c787-config-file" pod "db4c7c05-8a9f-4134-a0d7-69ee24d4c787" (UID: "db4c7c05-8a9f-4134-a0d7-69ee24d4c787"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:50:41.201071 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:41.201039 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4c7c05-8a9f-4134-a0d7-69ee24d4c787-kube-api-access-pbglj" pod "db4c7c05-8a9f-4134-a0d7-69ee24d4c787" (UID: "db4c7c05-8a9f-4134-a0d7-69ee24d4c787"). InnerVolumeSpecName "kube-api-access-pbglj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:50:41.218986 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:41.218954 5805 generic.go:356] "Generic (PLEG): container finished" podID="db4c7c05-8a9f-4134-a0d7-69ee24d4c787" containerID="1dd3779ff03049325e2444ea977b6a62bd07dcdb210630f19b915831f1696e93" exitCode=0 Nov 03 17:50:41.219114 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:41.219001 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9g5jz" Nov 03 17:50:41.219114 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:41.219011 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9g5jz" event={"ID":"db4c7c05-8a9f-4134-a0d7-69ee24d4c787","Type":"ContainerDied","Data":"1dd3779ff03049325e2444ea977b6a62bd07dcdb210630f19b915831f1696e93"} Nov 03 17:50:41.219114 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:41.219064 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9g5jz" event={"ID":"db4c7c05-8a9f-4134-a0d7-69ee24d4c787","Type":"ContainerDied","Data":"98f4c40a7c35866e515ea51518d3ed7b08b0b98e3fbc1ae8f18e634f6c73d952"} Nov 03 17:50:41.219114 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:41.219087 5805 scope.go:122] "RemoveContainer" containerID="1dd3779ff03049325e2444ea977b6a62bd07dcdb210630f19b915831f1696e93" Nov 03 17:50:41.228024 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:41.228005 5805 scope.go:122] "RemoveContainer" containerID="1dd3779ff03049325e2444ea977b6a62bd07dcdb210630f19b915831f1696e93" Nov 03 17:50:41.228304 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:50:41.228281 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dd3779ff03049325e2444ea977b6a62bd07dcdb210630f19b915831f1696e93\": container with ID starting with 1dd3779ff03049325e2444ea977b6a62bd07dcdb210630f19b915831f1696e93 not found: ID does not exist" containerID="1dd3779ff03049325e2444ea977b6a62bd07dcdb210630f19b915831f1696e93" Nov 03 17:50:41.228411 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:41.228312 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dd3779ff03049325e2444ea977b6a62bd07dcdb210630f19b915831f1696e93"} err="failed to get container status \"1dd3779ff03049325e2444ea977b6a62bd07dcdb210630f19b915831f1696e93\": rpc error: code = NotFound desc = could not find container \"1dd3779ff03049325e2444ea977b6a62bd07dcdb210630f19b915831f1696e93\": container with ID starting with 1dd3779ff03049325e2444ea977b6a62bd07dcdb210630f19b915831f1696e93 not found: ID does not exist" Nov 03 17:50:41.300495 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:41.300462 5805 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/db4c7c05-8a9f-4134-a0d7-69ee24d4c787-config-file\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:50:41.300495 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:41.300489 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbglj\" (UniqueName: \"kubernetes.io/projected/db4c7c05-8a9f-4134-a0d7-69ee24d4c787-kube-api-access-pbglj\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:50:41.373518 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:41.373481 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9g5jz"] Nov 03 17:50:41.378163 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:41.378133 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9g5jz"] Nov 03 17:50:42.974276 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:42.974202 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db4c7c05-8a9f-4134-a0d7-69ee24d4c787" path="/var/lib/kubelet/pods/db4c7c05-8a9f-4134-a0d7-69ee24d4c787/volumes" Nov 03 17:50:44.919027 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:44.918994 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:45.143859 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:45.143819 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:45.630663 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:45.630633 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-ff8wf"] Nov 03 17:50:45.630995 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:45.630983 5805 cpu_manager.go:414] "containerMap: removing container" podUID="db4c7c05-8a9f-4134-a0d7-69ee24d4c787" containerName="limitador" Nov 03 17:50:45.631039 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:45.630996 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4c7c05-8a9f-4134-a0d7-69ee24d4c787" containerName="limitador" Nov 03 17:50:45.631073 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:45.631052 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4c7c05-8a9f-4134-a0d7-69ee24d4c787" containerName="limitador" Nov 03 17:50:45.648233 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:45.648206 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ff8wf" Nov 03 17:50:45.689870 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:45.689840 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-ff8wf"] Nov 03 17:50:45.703638 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:45.703613 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 17:50:45.703638 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:45.703622 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:50:45.734104 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:45.734066 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k27fl\" (UniqueName: \"kubernetes.io/projected/e1e3a570-a9f9-41c6-b5ab-51898a697a90-kube-api-access-k27fl\") pod \"postgres-7cdbf6dc44-ff8wf\" (UID: \"e1e3a570-a9f9-41c6-b5ab-51898a697a90\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ff8wf" Nov 03 17:50:45.734278 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:45.734116 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e1e3a570-a9f9-41c6-b5ab-51898a697a90-data\") pod \"postgres-7cdbf6dc44-ff8wf\" (UID: \"e1e3a570-a9f9-41c6-b5ab-51898a697a90\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ff8wf" Nov 03 17:50:45.734784 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:45.734767 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-mllsl\"" Nov 03 17:50:45.734850 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:45.734810 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:50:45.811908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:45.811865 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:50:45.835098 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:45.835071 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k27fl\" (UniqueName: \"kubernetes.io/projected/e1e3a570-a9f9-41c6-b5ab-51898a697a90-kube-api-access-k27fl\") pod \"postgres-7cdbf6dc44-ff8wf\" (UID: \"e1e3a570-a9f9-41c6-b5ab-51898a697a90\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ff8wf" Nov 03 17:50:45.835263 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:45.835115 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e1e3a570-a9f9-41c6-b5ab-51898a697a90-data\") pod \"postgres-7cdbf6dc44-ff8wf\" (UID: \"e1e3a570-a9f9-41c6-b5ab-51898a697a90\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ff8wf" Nov 03 17:50:45.835513 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:45.835497 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e1e3a570-a9f9-41c6-b5ab-51898a697a90-data\") pod \"postgres-7cdbf6dc44-ff8wf\" (UID: \"e1e3a570-a9f9-41c6-b5ab-51898a697a90\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ff8wf" Nov 03 17:50:45.894300 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:45.894229 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k27fl\" (UniqueName: \"kubernetes.io/projected/e1e3a570-a9f9-41c6-b5ab-51898a697a90-kube-api-access-k27fl\") pod \"postgres-7cdbf6dc44-ff8wf\" (UID: \"e1e3a570-a9f9-41c6-b5ab-51898a697a90\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ff8wf" Nov 03 17:50:45.959958 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:45.959923 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ff8wf" Nov 03 17:50:46.325959 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:46.325927 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-ff8wf"] Nov 03 17:50:46.332960 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:50:46.332930 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e3a570_a9f9_41c6_b5ab_51898a697a90.slice/crio-bc6349b808419e7d4d82cf0ae0ae67d64a98e94e51853e85a2a067c23688e4c6 WatchSource:0}: Error finding container bc6349b808419e7d4d82cf0ae0ae67d64a98e94e51853e85a2a067c23688e4c6: Status 404 returned error can't find the container with id bc6349b808419e7d4d82cf0ae0ae67d64a98e94e51853e85a2a067c23688e4c6 Nov 03 17:50:47.248380 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:47.248347 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ff8wf" event={"ID":"e1e3a570-a9f9-41c6-b5ab-51898a697a90","Type":"ContainerStarted","Data":"bc6349b808419e7d4d82cf0ae0ae67d64a98e94e51853e85a2a067c23688e4c6"} Nov 03 17:50:47.693488 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:47.693454 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-sbsh4"] Nov 03 17:50:47.696985 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:47.696964 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbsh4" Nov 03 17:50:47.745786 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:47.745756 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 17:50:47.747312 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:47.747291 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/654e1963-654e-44e1-b284-1cf0bb09216d-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-sbsh4\" (UID: \"654e1963-654e-44e1-b284-1cf0bb09216d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbsh4" Nov 03 17:50:47.747380 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:47.747335 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/654e1963-654e-44e1-b284-1cf0bb09216d-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-sbsh4\" (UID: \"654e1963-654e-44e1-b284-1cf0bb09216d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbsh4" Nov 03 17:50:47.747444 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:47.747428 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvxxg\" (UniqueName: \"kubernetes.io/projected/654e1963-654e-44e1-b284-1cf0bb09216d-kube-api-access-kvxxg\") pod \"kserve-module-controller-manager-7566cf5fc8-sbsh4\" (UID: \"654e1963-654e-44e1-b284-1cf0bb09216d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbsh4" Nov 03 17:50:47.747499 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:47.747482 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/654e1963-654e-44e1-b284-1cf0bb09216d-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-sbsh4\" (UID: \"654e1963-654e-44e1-b284-1cf0bb09216d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbsh4" Nov 03 17:50:47.791955 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:47.791926 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 17:50:47.791955 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:47.791951 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-5msqt\"" Nov 03 17:50:47.848042 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:47.848002 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvxxg\" (UniqueName: \"kubernetes.io/projected/654e1963-654e-44e1-b284-1cf0bb09216d-kube-api-access-kvxxg\") pod \"kserve-module-controller-manager-7566cf5fc8-sbsh4\" (UID: \"654e1963-654e-44e1-b284-1cf0bb09216d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbsh4" Nov 03 17:50:47.848206 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:47.848054 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/654e1963-654e-44e1-b284-1cf0bb09216d-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-sbsh4\" (UID: \"654e1963-654e-44e1-b284-1cf0bb09216d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbsh4" Nov 03 17:50:47.848206 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:47.848093 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/654e1963-654e-44e1-b284-1cf0bb09216d-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-sbsh4\" (UID: \"654e1963-654e-44e1-b284-1cf0bb09216d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbsh4" Nov 03 17:50:47.848206 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:47.848146 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/654e1963-654e-44e1-b284-1cf0bb09216d-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-sbsh4\" (UID: \"654e1963-654e-44e1-b284-1cf0bb09216d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbsh4" Nov 03 17:50:47.848486 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:47.848469 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/654e1963-654e-44e1-b284-1cf0bb09216d-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-sbsh4\" (UID: \"654e1963-654e-44e1-b284-1cf0bb09216d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbsh4" Nov 03 17:50:47.848527 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:47.848500 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/654e1963-654e-44e1-b284-1cf0bb09216d-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-sbsh4\" (UID: \"654e1963-654e-44e1-b284-1cf0bb09216d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbsh4" Nov 03 17:50:47.885205 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:47.885174 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 17:50:47.886719 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:47.886702 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-sbsh4"] Nov 03 17:50:47.889943 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:47.889923 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/654e1963-654e-44e1-b284-1cf0bb09216d-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-sbsh4\" (UID: \"654e1963-654e-44e1-b284-1cf0bb09216d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbsh4" Nov 03 17:50:47.905814 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:47.905786 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvxxg\" (UniqueName: \"kubernetes.io/projected/654e1963-654e-44e1-b284-1cf0bb09216d-kube-api-access-kvxxg\") pod \"kserve-module-controller-manager-7566cf5fc8-sbsh4\" (UID: \"654e1963-654e-44e1-b284-1cf0bb09216d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbsh4" Nov 03 17:50:48.009835 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:48.009751 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbsh4" Nov 03 17:50:48.432529 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:48.432497 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-sbsh4"] Nov 03 17:50:48.439098 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:50:48.439068 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654e1963_654e_44e1_b284_1cf0bb09216d.slice/crio-f0373ac8390aab6577da14e270e471d128861fda23e0939cdcce7871e31aa7e4 WatchSource:0}: Error finding container f0373ac8390aab6577da14e270e471d128861fda23e0939cdcce7871e31aa7e4: Status 404 returned error can't find the container with id f0373ac8390aab6577da14e270e471d128861fda23e0939cdcce7871e31aa7e4 Nov 03 17:50:49.261351 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:49.261306 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbsh4" event={"ID":"654e1963-654e-44e1-b284-1cf0bb09216d","Type":"ContainerStarted","Data":"f0373ac8390aab6577da14e270e471d128861fda23e0939cdcce7871e31aa7e4"} Nov 03 17:50:51.276245 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:51.276200 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbsh4" event={"ID":"654e1963-654e-44e1-b284-1cf0bb09216d","Type":"ContainerStarted","Data":"1d7054d22f17e77e255da4915999e0465e91d31e10b22c4770ee07ada53193f3"} Nov 03 17:50:51.276729 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:51.276305 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbsh4" Nov 03 17:50:51.455130 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:51.455061 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbsh4" podStartSLOduration=2.133089158 podStartE2EDuration="4.455039721s" totalImagesPullingTime="2.321950563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:50:51.453964344 +0000 UTC m=+288.587593379" watchObservedRunningTime="2026-11-03 17:50:51.455039721 +0000 UTC m=+288.588668732" Nov 03 17:50:53.288545 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:53.288509 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ff8wf" event={"ID":"e1e3a570-a9f9-41c6-b5ab-51898a697a90","Type":"ContainerStarted","Data":"b3b12ddc9a2476e3a518a0f194a78cbabbfafae962c3972d0c0b2d09b65b6611"} Nov 03 17:50:53.288892 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:53.288578 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ff8wf" Nov 03 17:50:53.399462 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:53.399384 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ff8wf" podStartSLOduration=1.537729839 podStartE2EDuration="8.399367396s" totalImagesPullingTime="6.861637557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:50:53.398347547 +0000 UTC m=+290.531976558" watchObservedRunningTime="2026-11-03 17:50:53.399367396 +0000 UTC m=+290.532996407" Nov 03 17:50:56.234762 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:50:56.234721 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:50:59.321771 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:50:59.321739 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ff8wf" Nov 03 17:51:03.020066 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:03.020035 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 17:51:03.020725 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:03.020708 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 17:51:06.609289 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:06.609252 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:51:06.690582 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:06.690554 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9pts2"] Nov 03 17:51:06.697937 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:06.697914 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-9pts2" Nov 03 17:51:06.746733 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:06.746702 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 17:51:06.792591 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:06.792551 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 17:51:06.792591 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:06.792557 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kcmdt\"" Nov 03 17:51:06.792783 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:06.792699 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 17:51:06.794245 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:06.794225 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9pts2"] Nov 03 17:51:06.814957 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:06.814936 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5f41a03-fc6a-4968-9dd1-f082a3d6001d-metrics-certs\") pod \"maas-controller-579cf7c6d-9pts2\" (UID: \"f5f41a03-fc6a-4968-9dd1-f082a3d6001d\") " pod="opendatahub/maas-controller-579cf7c6d-9pts2" Nov 03 17:51:06.815053 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:06.814980 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f5f41a03-fc6a-4968-9dd1-f082a3d6001d-webhook-cert\") pod \"maas-controller-579cf7c6d-9pts2\" (UID: \"f5f41a03-fc6a-4968-9dd1-f082a3d6001d\") " pod="opendatahub/maas-controller-579cf7c6d-9pts2" Nov 03 17:51:06.815053 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:06.814999 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhng7\" (UniqueName: \"kubernetes.io/projected/f5f41a03-fc6a-4968-9dd1-f082a3d6001d-kube-api-access-vhng7\") pod \"maas-controller-579cf7c6d-9pts2\" (UID: \"f5f41a03-fc6a-4968-9dd1-f082a3d6001d\") " pod="opendatahub/maas-controller-579cf7c6d-9pts2" Nov 03 17:51:06.819513 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:06.819490 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:51:06.915623 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:06.915568 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f5f41a03-fc6a-4968-9dd1-f082a3d6001d-webhook-cert\") pod \"maas-controller-579cf7c6d-9pts2\" (UID: \"f5f41a03-fc6a-4968-9dd1-f082a3d6001d\") " pod="opendatahub/maas-controller-579cf7c6d-9pts2" Nov 03 17:51:06.915623 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:06.915598 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhng7\" (UniqueName: \"kubernetes.io/projected/f5f41a03-fc6a-4968-9dd1-f082a3d6001d-kube-api-access-vhng7\") pod \"maas-controller-579cf7c6d-9pts2\" (UID: \"f5f41a03-fc6a-4968-9dd1-f082a3d6001d\") " pod="opendatahub/maas-controller-579cf7c6d-9pts2" Nov 03 17:51:06.915797 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:06.915660 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5f41a03-fc6a-4968-9dd1-f082a3d6001d-metrics-certs\") pod \"maas-controller-579cf7c6d-9pts2\" (UID: \"f5f41a03-fc6a-4968-9dd1-f082a3d6001d\") " pod="opendatahub/maas-controller-579cf7c6d-9pts2" Nov 03 17:51:06.918076 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:06.918056 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5f41a03-fc6a-4968-9dd1-f082a3d6001d-metrics-certs\") pod \"maas-controller-579cf7c6d-9pts2\" (UID: \"f5f41a03-fc6a-4968-9dd1-f082a3d6001d\") " pod="opendatahub/maas-controller-579cf7c6d-9pts2" Nov 03 17:51:06.918163 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:06.918118 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f5f41a03-fc6a-4968-9dd1-f082a3d6001d-webhook-cert\") pod \"maas-controller-579cf7c6d-9pts2\" (UID: \"f5f41a03-fc6a-4968-9dd1-f082a3d6001d\") " pod="opendatahub/maas-controller-579cf7c6d-9pts2" Nov 03 17:51:06.973368 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:06.973345 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhng7\" (UniqueName: \"kubernetes.io/projected/f5f41a03-fc6a-4968-9dd1-f082a3d6001d-kube-api-access-vhng7\") pod \"maas-controller-579cf7c6d-9pts2\" (UID: \"f5f41a03-fc6a-4968-9dd1-f082a3d6001d\") " pod="opendatahub/maas-controller-579cf7c6d-9pts2" Nov 03 17:51:07.010242 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:07.010224 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-9pts2" Nov 03 17:51:07.023770 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:07.023745 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:51:07.109914 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:07.109887 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c8b7c4cfc-4lkmm"] Nov 03 17:51:07.113355 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:07.113335 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c8b7c4cfc-4lkmm" Nov 03 17:51:07.170794 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:07.170739 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c8b7c4cfc-4lkmm"] Nov 03 17:51:07.218722 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:07.218697 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60dc8570-85f6-4133-9567-5e99139b44e7-metrics-certs\") pod \"maas-controller-5c8b7c4cfc-4lkmm\" (UID: \"60dc8570-85f6-4133-9567-5e99139b44e7\") " pod="opendatahub/maas-controller-5c8b7c4cfc-4lkmm" Nov 03 17:51:07.218837 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:07.218726 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvg4z\" (UniqueName: \"kubernetes.io/projected/60dc8570-85f6-4133-9567-5e99139b44e7-kube-api-access-fvg4z\") pod \"maas-controller-5c8b7c4cfc-4lkmm\" (UID: \"60dc8570-85f6-4133-9567-5e99139b44e7\") " pod="opendatahub/maas-controller-5c8b7c4cfc-4lkmm" Nov 03 17:51:07.218887 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:07.218841 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60dc8570-85f6-4133-9567-5e99139b44e7-webhook-cert\") pod \"maas-controller-5c8b7c4cfc-4lkmm\" (UID: \"60dc8570-85f6-4133-9567-5e99139b44e7\") " pod="opendatahub/maas-controller-5c8b7c4cfc-4lkmm" Nov 03 17:51:07.233170 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:07.233145 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:51:07.320316 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:07.320289 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60dc8570-85f6-4133-9567-5e99139b44e7-webhook-cert\") pod \"maas-controller-5c8b7c4cfc-4lkmm\" (UID: \"60dc8570-85f6-4133-9567-5e99139b44e7\") " pod="opendatahub/maas-controller-5c8b7c4cfc-4lkmm" Nov 03 17:51:07.320476 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:07.320330 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60dc8570-85f6-4133-9567-5e99139b44e7-metrics-certs\") pod \"maas-controller-5c8b7c4cfc-4lkmm\" (UID: \"60dc8570-85f6-4133-9567-5e99139b44e7\") " pod="opendatahub/maas-controller-5c8b7c4cfc-4lkmm" Nov 03 17:51:07.320476 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:07.320361 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvg4z\" (UniqueName: \"kubernetes.io/projected/60dc8570-85f6-4133-9567-5e99139b44e7-kube-api-access-fvg4z\") pod \"maas-controller-5c8b7c4cfc-4lkmm\" (UID: \"60dc8570-85f6-4133-9567-5e99139b44e7\") " pod="opendatahub/maas-controller-5c8b7c4cfc-4lkmm" Nov 03 17:51:07.322798 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:07.322777 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60dc8570-85f6-4133-9567-5e99139b44e7-webhook-cert\") pod \"maas-controller-5c8b7c4cfc-4lkmm\" (UID: \"60dc8570-85f6-4133-9567-5e99139b44e7\") " pod="opendatahub/maas-controller-5c8b7c4cfc-4lkmm" Nov 03 17:51:07.322893 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:07.322816 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60dc8570-85f6-4133-9567-5e99139b44e7-metrics-certs\") pod \"maas-controller-5c8b7c4cfc-4lkmm\" (UID: \"60dc8570-85f6-4133-9567-5e99139b44e7\") " pod="opendatahub/maas-controller-5c8b7c4cfc-4lkmm" Nov 03 17:51:07.378223 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:07.378203 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvg4z\" (UniqueName: \"kubernetes.io/projected/60dc8570-85f6-4133-9567-5e99139b44e7-kube-api-access-fvg4z\") pod \"maas-controller-5c8b7c4cfc-4lkmm\" (UID: \"60dc8570-85f6-4133-9567-5e99139b44e7\") " pod="opendatahub/maas-controller-5c8b7c4cfc-4lkmm" Nov 03 17:51:07.382273 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:07.382252 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9pts2"] Nov 03 17:51:07.388987 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:51:07.388961 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f41a03_fc6a_4968_9dd1_f082a3d6001d.slice/crio-263e6dd404a3544a42881714c6fd1027368d2662893a19a340b9e32637d55dcd WatchSource:0}: Error finding container 263e6dd404a3544a42881714c6fd1027368d2662893a19a340b9e32637d55dcd: Status 404 returned error can't find the container with id 263e6dd404a3544a42881714c6fd1027368d2662893a19a340b9e32637d55dcd Nov 03 17:51:07.425291 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:07.425232 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c8b7c4cfc-4lkmm" Nov 03 17:51:07.797906 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:07.797830 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c8b7c4cfc-4lkmm"] Nov 03 17:51:07.803340 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:51:07.803314 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60dc8570_85f6_4133_9567_5e99139b44e7.slice/crio-cf299eee582e7c015aed7f8ce0f5eb6d358bdc718fc34bc1616857c8e087c58e WatchSource:0}: Error finding container cf299eee582e7c015aed7f8ce0f5eb6d358bdc718fc34bc1616857c8e087c58e: Status 404 returned error can't find the container with id cf299eee582e7c015aed7f8ce0f5eb6d358bdc718fc34bc1616857c8e087c58e Nov 03 17:51:08.358739 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:08.358702 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-9pts2" event={"ID":"f5f41a03-fc6a-4968-9dd1-f082a3d6001d","Type":"ContainerStarted","Data":"263e6dd404a3544a42881714c6fd1027368d2662893a19a340b9e32637d55dcd"} Nov 03 17:51:08.359921 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:08.359890 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c8b7c4cfc-4lkmm" event={"ID":"60dc8570-85f6-4133-9567-5e99139b44e7","Type":"ContainerStarted","Data":"cf299eee582e7c015aed7f8ce0f5eb6d358bdc718fc34bc1616857c8e087c58e"} Nov 03 17:51:11.382063 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:11.382024 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-9pts2" event={"ID":"f5f41a03-fc6a-4968-9dd1-f082a3d6001d","Type":"ContainerStarted","Data":"5e99ab8d2480e6a5b7d196181d7ea18324b418de190fb074f5f98799fe33f55e"} Nov 03 17:51:11.382505 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:11.382095 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-9pts2" Nov 03 17:51:11.383795 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:11.383767 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c8b7c4cfc-4lkmm" event={"ID":"60dc8570-85f6-4133-9567-5e99139b44e7","Type":"ContainerStarted","Data":"341194b77799fcd79668dcdf981c3d8d0e8c27f3a5d5f6cc8a0086f3f2a00fc9"} Nov 03 17:51:11.383957 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:11.383857 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c8b7c4cfc-4lkmm" Nov 03 17:51:11.510716 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:11.510670 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:51:11.513309 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:11.513263 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-9pts2" podStartSLOduration=1.993089523 podStartE2EDuration="5.513249575s" totalImagesPullingTime="3.520160052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:51:11.511992905 +0000 UTC m=+308.645621945" watchObservedRunningTime="2026-11-03 17:51:11.513249575 +0000 UTC m=+308.646878612" Nov 03 17:51:11.637960 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:11.637878 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5c8b7c4cfc-4lkmm" podStartSLOduration=1.343187617 podStartE2EDuration="4.637864757s" totalImagesPullingTime="3.29467714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:51:11.636779768 +0000 UTC m=+308.770408789" watchObservedRunningTime="2026-11-03 17:51:11.637864757 +0000 UTC m=+308.771493766" Nov 03 17:51:11.767358 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:11.767315 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:51:12.282101 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:12.282072 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-sbsh4" Nov 03 17:51:14.332331 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:14.332302 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b9mh2"] Nov 03 17:51:14.335672 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:14.335651 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b9mh2" Nov 03 17:51:14.385291 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:14.385270 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 03 17:51:14.385378 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:14.385292 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 03 17:51:14.386966 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:14.386948 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnt84\" (UniqueName: \"kubernetes.io/projected/f544ec2f-3048-4b83-a594-0c7a412c3076-kube-api-access-vnt84\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b9mh2\" (UID: \"f544ec2f-3048-4b83-a594-0c7a412c3076\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b9mh2" Nov 03 17:51:14.387031 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:14.386977 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f544ec2f-3048-4b83-a594-0c7a412c3076-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b9mh2\" (UID: \"f544ec2f-3048-4b83-a594-0c7a412c3076\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b9mh2" Nov 03 17:51:14.387031 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:14.387017 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f544ec2f-3048-4b83-a594-0c7a412c3076-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b9mh2\" (UID: \"f544ec2f-3048-4b83-a594-0c7a412c3076\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b9mh2" Nov 03 17:51:14.387191 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:14.387175 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f544ec2f-3048-4b83-a594-0c7a412c3076-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b9mh2\" (UID: \"f544ec2f-3048-4b83-a594-0c7a412c3076\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b9mh2" Nov 03 17:51:14.431625 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:14.431575 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 03 17:51:14.431774 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:14.431616 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-z7hm8\"" Nov 03 17:51:14.433424 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:14.433379 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b9mh2"] Nov 03 17:51:14.488280 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:14.488254 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f544ec2f-3048-4b83-a594-0c7a412c3076-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b9mh2\" (UID: \"f544ec2f-3048-4b83-a594-0c7a412c3076\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b9mh2" Nov 03 17:51:14.488448 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:14.488300 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnt84\" (UniqueName: \"kubernetes.io/projected/f544ec2f-3048-4b83-a594-0c7a412c3076-kube-api-access-vnt84\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b9mh2\" (UID: \"f544ec2f-3048-4b83-a594-0c7a412c3076\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b9mh2" Nov 03 17:51:14.488448 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:14.488353 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f544ec2f-3048-4b83-a594-0c7a412c3076-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b9mh2\" (UID: \"f544ec2f-3048-4b83-a594-0c7a412c3076\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b9mh2" Nov 03 17:51:14.488448 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:51:14.488375 5805 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 03 17:51:14.488448 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:14.488432 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f544ec2f-3048-4b83-a594-0c7a412c3076-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b9mh2\" (UID: \"f544ec2f-3048-4b83-a594-0c7a412c3076\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b9mh2" Nov 03 17:51:14.488674 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:51:14.488463 5805 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f544ec2f-3048-4b83-a594-0c7a412c3076-proxy-source podName:f544ec2f-3048-4b83-a594-0c7a412c3076 nodeName:}" failed. No retries permitted until 2026-11-03 17:51:14.988441675 +0000 UTC m=+312.122070691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/f544ec2f-3048-4b83-a594-0c7a412c3076-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-b9mh2" (UID: "f544ec2f-3048-4b83-a594-0c7a412c3076") : configmap "usage-tenancy-proxy-source" not found Nov 03 17:51:14.488674 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:51:14.488533 5805 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Nov 03 17:51:14.488674 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:51:14.488595 5805 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f544ec2f-3048-4b83-a594-0c7a412c3076-tls-cert podName:f544ec2f-3048-4b83-a594-0c7a412c3076 nodeName:}" failed. No retries permitted until 2026-11-03 17:51:14.988577413 +0000 UTC m=+312.122206425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/f544ec2f-3048-4b83-a594-0c7a412c3076-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-b9mh2" (UID: "f544ec2f-3048-4b83-a594-0c7a412c3076") : secret "usage-logs-tenancy-proxy-tls" not found Nov 03 17:51:14.489019 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:14.489002 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f544ec2f-3048-4b83-a594-0c7a412c3076-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b9mh2\" (UID: \"f544ec2f-3048-4b83-a594-0c7a412c3076\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b9mh2" Nov 03 17:51:14.542408 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:14.542346 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:51:14.547803 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:14.547785 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnt84\" (UniqueName: \"kubernetes.io/projected/f544ec2f-3048-4b83-a594-0c7a412c3076-kube-api-access-vnt84\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b9mh2\" (UID: \"f544ec2f-3048-4b83-a594-0c7a412c3076\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b9mh2" Nov 03 17:51:14.791026 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:14.790958 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:51:14.992377 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:14.992337 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f544ec2f-3048-4b83-a594-0c7a412c3076-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b9mh2\" (UID: \"f544ec2f-3048-4b83-a594-0c7a412c3076\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b9mh2" Nov 03 17:51:14.992551 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:14.992414 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f544ec2f-3048-4b83-a594-0c7a412c3076-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b9mh2\" (UID: \"f544ec2f-3048-4b83-a594-0c7a412c3076\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b9mh2" Nov 03 17:51:14.993262 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:14.993238 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f544ec2f-3048-4b83-a594-0c7a412c3076-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b9mh2\" (UID: \"f544ec2f-3048-4b83-a594-0c7a412c3076\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b9mh2" Nov 03 17:51:14.994866 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:14.994844 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f544ec2f-3048-4b83-a594-0c7a412c3076-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b9mh2\" (UID: \"f544ec2f-3048-4b83-a594-0c7a412c3076\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b9mh2" Nov 03 17:51:15.247299 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:15.247267 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b9mh2" Nov 03 17:51:15.617229 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:15.617199 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b9mh2"] Nov 03 17:51:15.624570 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:51:15.624543 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf544ec2f_3048_4b83_a594_0c7a412c3076.slice/crio-dd040188f65cf9d96ecb940ca38f1b29796f291e8e06ed03295dd04d106401bd WatchSource:0}: Error finding container dd040188f65cf9d96ecb940ca38f1b29796f291e8e06ed03295dd04d106401bd: Status 404 returned error can't find the container with id dd040188f65cf9d96ecb940ca38f1b29796f291e8e06ed03295dd04d106401bd Nov 03 17:51:16.408313 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:16.408276 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b9mh2" event={"ID":"f544ec2f-3048-4b83-a594-0c7a412c3076","Type":"ContainerStarted","Data":"dd040188f65cf9d96ecb940ca38f1b29796f291e8e06ed03295dd04d106401bd"} Nov 03 17:51:22.390422 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:22.390367 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-9pts2" Nov 03 17:51:22.390915 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:22.390492 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c8b7c4cfc-4lkmm" Nov 03 17:51:22.648166 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:22.648055 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9pts2"] Nov 03 17:51:22.648791 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:22.648618 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-9pts2" podUID="f5f41a03-fc6a-4968-9dd1-f082a3d6001d" containerName="manager" containerID="cri-o://5e99ab8d2480e6a5b7d196181d7ea18324b418de190fb074f5f98799fe33f55e" gracePeriod=30 Nov 03 17:51:22.950200 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:22.950171 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-9pts2" Nov 03 17:51:23.082730 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:23.082688 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5f41a03-fc6a-4968-9dd1-f082a3d6001d-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f5f41a03-fc6a-4968-9dd1-f082a3d6001d-webhook-cert\") pod \"f5f41a03-fc6a-4968-9dd1-f082a3d6001d\" (UID: \"f5f41a03-fc6a-4968-9dd1-f082a3d6001d\") " Nov 03 17:51:23.082910 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:23.082755 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5f41a03-fc6a-4968-9dd1-f082a3d6001d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5f41a03-fc6a-4968-9dd1-f082a3d6001d-metrics-certs\") pod \"f5f41a03-fc6a-4968-9dd1-f082a3d6001d\" (UID: \"f5f41a03-fc6a-4968-9dd1-f082a3d6001d\") " Nov 03 17:51:23.083241 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:23.082898 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5f41a03-fc6a-4968-9dd1-f082a3d6001d-kube-api-access-vhng7\" (UniqueName: \"kubernetes.io/projected/f5f41a03-fc6a-4968-9dd1-f082a3d6001d-kube-api-access-vhng7\") pod \"f5f41a03-fc6a-4968-9dd1-f082a3d6001d\" (UID: \"f5f41a03-fc6a-4968-9dd1-f082a3d6001d\") " Nov 03 17:51:23.085451 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:23.085362 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5f41a03-fc6a-4968-9dd1-f082a3d6001d-metrics-certs" pod "f5f41a03-fc6a-4968-9dd1-f082a3d6001d" (UID: "f5f41a03-fc6a-4968-9dd1-f082a3d6001d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:51:23.085561 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:23.085466 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f41a03-fc6a-4968-9dd1-f082a3d6001d-kube-api-access-vhng7" pod "f5f41a03-fc6a-4968-9dd1-f082a3d6001d" (UID: "f5f41a03-fc6a-4968-9dd1-f082a3d6001d"). InnerVolumeSpecName "kube-api-access-vhng7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:51:23.085561 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:23.085502 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5f41a03-fc6a-4968-9dd1-f082a3d6001d-webhook-cert" pod "f5f41a03-fc6a-4968-9dd1-f082a3d6001d" (UID: "f5f41a03-fc6a-4968-9dd1-f082a3d6001d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:51:23.184077 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:23.184002 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhng7\" (UniqueName: \"kubernetes.io/projected/f5f41a03-fc6a-4968-9dd1-f082a3d6001d-kube-api-access-vhng7\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:51:23.184077 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:23.184033 5805 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f5f41a03-fc6a-4968-9dd1-f082a3d6001d-webhook-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:51:23.184077 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:23.184043 5805 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5f41a03-fc6a-4968-9dd1-f082a3d6001d-metrics-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:51:23.444719 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:23.444624 5805 generic.go:356] "Generic (PLEG): container finished" podID="f5f41a03-fc6a-4968-9dd1-f082a3d6001d" containerID="5e99ab8d2480e6a5b7d196181d7ea18324b418de190fb074f5f98799fe33f55e" exitCode=0 Nov 03 17:51:23.444719 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:23.444671 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-9pts2" Nov 03 17:51:23.444719 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:23.444703 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-9pts2" event={"ID":"f5f41a03-fc6a-4968-9dd1-f082a3d6001d","Type":"ContainerDied","Data":"5e99ab8d2480e6a5b7d196181d7ea18324b418de190fb074f5f98799fe33f55e"} Nov 03 17:51:23.445428 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:23.444754 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-9pts2" event={"ID":"f5f41a03-fc6a-4968-9dd1-f082a3d6001d","Type":"ContainerDied","Data":"263e6dd404a3544a42881714c6fd1027368d2662893a19a340b9e32637d55dcd"} Nov 03 17:51:23.445428 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:23.444785 5805 scope.go:122] "RemoveContainer" containerID="5e99ab8d2480e6a5b7d196181d7ea18324b418de190fb074f5f98799fe33f55e" Nov 03 17:51:23.455753 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:23.455730 5805 scope.go:122] "RemoveContainer" containerID="5e99ab8d2480e6a5b7d196181d7ea18324b418de190fb074f5f98799fe33f55e" Nov 03 17:51:23.456030 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:51:23.456010 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e99ab8d2480e6a5b7d196181d7ea18324b418de190fb074f5f98799fe33f55e\": container with ID starting with 5e99ab8d2480e6a5b7d196181d7ea18324b418de190fb074f5f98799fe33f55e not found: ID does not exist" containerID="5e99ab8d2480e6a5b7d196181d7ea18324b418de190fb074f5f98799fe33f55e" Nov 03 17:51:23.456094 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:23.456038 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e99ab8d2480e6a5b7d196181d7ea18324b418de190fb074f5f98799fe33f55e"} err="failed to get container status \"5e99ab8d2480e6a5b7d196181d7ea18324b418de190fb074f5f98799fe33f55e\": rpc error: code = NotFound desc = could not find container \"5e99ab8d2480e6a5b7d196181d7ea18324b418de190fb074f5f98799fe33f55e\": container with ID starting with 5e99ab8d2480e6a5b7d196181d7ea18324b418de190fb074f5f98799fe33f55e not found: ID does not exist" Nov 03 17:51:23.599793 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:23.599758 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9pts2"] Nov 03 17:51:23.603677 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:23.603644 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-9pts2"] Nov 03 17:51:24.975765 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:24.975727 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5f41a03-fc6a-4968-9dd1-f082a3d6001d" path="/var/lib/kubelet/pods/f5f41a03-fc6a-4968-9dd1-f082a3d6001d/volumes" Nov 03 17:51:26.236209 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:51:26.236171 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:51:32.494247 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:32.494169 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b9mh2" event={"ID":"f544ec2f-3048-4b83-a594-0c7a412c3076","Type":"ContainerStarted","Data":"12180d24d38932dc714f5c6fa22d62800e34074245e1c0729332f15d2ceddd99"} Nov 03 17:51:32.494658 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:32.494281 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b9mh2" Nov 03 17:51:32.628112 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:32.628055 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b9mh2" podStartSLOduration=2.19404575 podStartE2EDuration="18.62804143s" totalImagesPullingTime="16.43399568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:51:32.626370545 +0000 UTC m=+329.759999572" watchObservedRunningTime="2026-11-03 17:51:32.62804143 +0000 UTC m=+329.761670439" Nov 03 17:51:43.504180 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:43.504145 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b9mh2" Nov 03 17:51:44.877673 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:44.877640 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:51:45.101919 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:45.101885 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:51:46.912616 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:46.912560 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:51:47.163211 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:47.163133 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:51:47.379438 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:47.379381 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:51:47.997491 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:47.997456 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5"] Nov 03 17:51:47.997983 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:47.997850 5805 cpu_manager.go:414] "containerMap: removing container" podUID="f5f41a03-fc6a-4968-9dd1-f082a3d6001d" containerName="manager" Nov 03 17:51:47.997983 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:47.997865 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5f41a03-fc6a-4968-9dd1-f082a3d6001d" containerName="manager" Nov 03 17:51:47.997983 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:47.997943 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5f41a03-fc6a-4968-9dd1-f082a3d6001d" containerName="manager" Nov 03 17:51:48.002770 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:48.002750 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5" Nov 03 17:51:48.098016 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:48.097990 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 17:51:48.098186 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:48.098091 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tr54c\"" Nov 03 17:51:48.098186 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:48.098144 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 17:51:48.099774 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:48.099755 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5"] Nov 03 17:51:48.129835 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:48.129808 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e948b450-b138-4a11-b189-56d45a657463-metrics-certs\") pod \"maas-api-6d669c7fcd-vzfx5\" (UID: \"e948b450-b138-4a11-b189-56d45a657463\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5" Nov 03 17:51:48.129991 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:48.129848 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvp6d\" (UniqueName: \"kubernetes.io/projected/e948b450-b138-4a11-b189-56d45a657463-kube-api-access-xvp6d\") pod \"maas-api-6d669c7fcd-vzfx5\" (UID: \"e948b450-b138-4a11-b189-56d45a657463\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5" Nov 03 17:51:48.129991 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:48.129946 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e948b450-b138-4a11-b189-56d45a657463-openshift-service-ca\") pod \"maas-api-6d669c7fcd-vzfx5\" (UID: \"e948b450-b138-4a11-b189-56d45a657463\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5" Nov 03 17:51:48.130101 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:48.130002 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e948b450-b138-4a11-b189-56d45a657463-maas-api-tls\") pod \"maas-api-6d669c7fcd-vzfx5\" (UID: \"e948b450-b138-4a11-b189-56d45a657463\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5" Nov 03 17:51:48.231364 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:48.231340 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e948b450-b138-4a11-b189-56d45a657463-metrics-certs\") pod \"maas-api-6d669c7fcd-vzfx5\" (UID: \"e948b450-b138-4a11-b189-56d45a657463\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5" Nov 03 17:51:48.231530 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:48.231384 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvp6d\" (UniqueName: \"kubernetes.io/projected/e948b450-b138-4a11-b189-56d45a657463-kube-api-access-xvp6d\") pod \"maas-api-6d669c7fcd-vzfx5\" (UID: \"e948b450-b138-4a11-b189-56d45a657463\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5" Nov 03 17:51:48.231530 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:48.231468 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e948b450-b138-4a11-b189-56d45a657463-openshift-service-ca\") pod \"maas-api-6d669c7fcd-vzfx5\" (UID: \"e948b450-b138-4a11-b189-56d45a657463\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5" Nov 03 17:51:48.231530 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:48.231505 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e948b450-b138-4a11-b189-56d45a657463-maas-api-tls\") pod \"maas-api-6d669c7fcd-vzfx5\" (UID: \"e948b450-b138-4a11-b189-56d45a657463\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5" Nov 03 17:51:48.232043 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:48.232023 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e948b450-b138-4a11-b189-56d45a657463-openshift-service-ca\") pod \"maas-api-6d669c7fcd-vzfx5\" (UID: \"e948b450-b138-4a11-b189-56d45a657463\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5" Nov 03 17:51:48.233759 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:48.233736 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e948b450-b138-4a11-b189-56d45a657463-maas-api-tls\") pod \"maas-api-6d669c7fcd-vzfx5\" (UID: \"e948b450-b138-4a11-b189-56d45a657463\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5" Nov 03 17:51:48.233838 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:48.233769 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e948b450-b138-4a11-b189-56d45a657463-metrics-certs\") pod \"maas-api-6d669c7fcd-vzfx5\" (UID: \"e948b450-b138-4a11-b189-56d45a657463\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5" Nov 03 17:51:48.288542 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:48.288489 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvp6d\" (UniqueName: \"kubernetes.io/projected/e948b450-b138-4a11-b189-56d45a657463-kube-api-access-xvp6d\") pod \"maas-api-6d669c7fcd-vzfx5\" (UID: \"e948b450-b138-4a11-b189-56d45a657463\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5" Nov 03 17:51:48.314485 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:48.314464 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5" Nov 03 17:51:48.689031 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:48.688999 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5"] Nov 03 17:51:48.695527 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:51:48.695493 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode948b450_b138_4a11_b189_56d45a657463.slice/crio-d3f623b34c2b867da4a494d6dd4f94e2fa929dc7f07aafc94b5f29590b47e029 WatchSource:0}: Error finding container d3f623b34c2b867da4a494d6dd4f94e2fa929dc7f07aafc94b5f29590b47e029: Status 404 returned error can't find the container with id d3f623b34c2b867da4a494d6dd4f94e2fa929dc7f07aafc94b5f29590b47e029 Nov 03 17:51:49.574806 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:49.574775 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5" event={"ID":"e948b450-b138-4a11-b189-56d45a657463","Type":"ContainerStarted","Data":"d3f623b34c2b867da4a494d6dd4f94e2fa929dc7f07aafc94b5f29590b47e029"} Nov 03 17:51:54.600987 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:54.600945 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5" event={"ID":"e948b450-b138-4a11-b189-56d45a657463","Type":"ContainerStarted","Data":"5c563de9a2a83ba2011d20f5a43a21df3a0c9492038100eaaa738dea22c892d0"} Nov 03 17:51:54.601471 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:54.601059 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5" Nov 03 17:51:54.714333 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:54.714278 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5" podStartSLOduration=2.722493273 podStartE2EDuration="7.714266265s" totalImagesPullingTime="4.991772992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:51:54.713388035 +0000 UTC m=+351.847017055" watchObservedRunningTime="2026-11-03 17:51:54.714266265 +0000 UTC m=+351.847895275" Nov 03 17:51:55.330364 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:55.330331 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:51:55.532174 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:55.532142 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:51:55.727668 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:55.727601 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:51:55.973198 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:55.973167 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:51:56.179473 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.179439 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:51:56.235232 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:51:56.235200 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:51:56.285240 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.285214 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qtwgk"] Nov 03 17:51:56.288062 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.288043 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qtwgk" Nov 03 17:51:56.336978 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.336953 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 03 17:51:56.336978 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.336953 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-jwznm\"" Nov 03 17:51:56.376210 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.376180 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:51:56.384268 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.384243 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qtwgk"] Nov 03 17:51:56.401442 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.401418 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-q7657"] Nov 03 17:51:56.404140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.404123 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-q7657" Nov 03 17:51:56.409549 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.409525 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ca0d7ec-0884-4202-af1f-e49a18e4f0ff-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-qtwgk\" (UID: \"2ca0d7ec-0884-4202-af1f-e49a18e4f0ff\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qtwgk" Nov 03 17:51:56.409638 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.409586 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr86k\" (UniqueName: \"kubernetes.io/projected/2ca0d7ec-0884-4202-af1f-e49a18e4f0ff-kube-api-access-sr86k\") pod \"llmisvc-controller-manager-869d5dcc5d-qtwgk\" (UID: \"2ca0d7ec-0884-4202-af1f-e49a18e4f0ff\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qtwgk" Nov 03 17:51:56.453178 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.453067 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 03 17:51:56.453178 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.453070 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-slcwd\"" Nov 03 17:51:56.461890 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.461856 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-q7657"] Nov 03 17:51:56.510130 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.510098 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/707bc0f1-d6de-45c3-9783-dd6772ca19eb-tls-certs\") pod \"model-serving-api-75bd89fdfc-q7657\" (UID: \"707bc0f1-d6de-45c3-9783-dd6772ca19eb\") " pod="opendatahub/model-serving-api-75bd89fdfc-q7657" Nov 03 17:51:56.510307 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.510141 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxtv5\" (UniqueName: \"kubernetes.io/projected/707bc0f1-d6de-45c3-9783-dd6772ca19eb-kube-api-access-rxtv5\") pod \"model-serving-api-75bd89fdfc-q7657\" (UID: \"707bc0f1-d6de-45c3-9783-dd6772ca19eb\") " pod="opendatahub/model-serving-api-75bd89fdfc-q7657" Nov 03 17:51:56.510307 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.510265 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ca0d7ec-0884-4202-af1f-e49a18e4f0ff-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-qtwgk\" (UID: \"2ca0d7ec-0884-4202-af1f-e49a18e4f0ff\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qtwgk" Nov 03 17:51:56.510465 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.510317 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr86k\" (UniqueName: \"kubernetes.io/projected/2ca0d7ec-0884-4202-af1f-e49a18e4f0ff-kube-api-access-sr86k\") pod \"llmisvc-controller-manager-869d5dcc5d-qtwgk\" (UID: \"2ca0d7ec-0884-4202-af1f-e49a18e4f0ff\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qtwgk" Nov 03 17:51:56.513192 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.513156 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ca0d7ec-0884-4202-af1f-e49a18e4f0ff-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-qtwgk\" (UID: \"2ca0d7ec-0884-4202-af1f-e49a18e4f0ff\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qtwgk" Nov 03 17:51:56.568045 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.568015 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr86k\" (UniqueName: \"kubernetes.io/projected/2ca0d7ec-0884-4202-af1f-e49a18e4f0ff-kube-api-access-sr86k\") pod \"llmisvc-controller-manager-869d5dcc5d-qtwgk\" (UID: \"2ca0d7ec-0884-4202-af1f-e49a18e4f0ff\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qtwgk" Nov 03 17:51:56.599695 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.599664 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qtwgk" Nov 03 17:51:56.611219 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.611188 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/707bc0f1-d6de-45c3-9783-dd6772ca19eb-tls-certs\") pod \"model-serving-api-75bd89fdfc-q7657\" (UID: \"707bc0f1-d6de-45c3-9783-dd6772ca19eb\") " pod="opendatahub/model-serving-api-75bd89fdfc-q7657" Nov 03 17:51:56.611357 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.611229 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxtv5\" (UniqueName: \"kubernetes.io/projected/707bc0f1-d6de-45c3-9783-dd6772ca19eb-kube-api-access-rxtv5\") pod \"model-serving-api-75bd89fdfc-q7657\" (UID: \"707bc0f1-d6de-45c3-9783-dd6772ca19eb\") " pod="opendatahub/model-serving-api-75bd89fdfc-q7657" Nov 03 17:51:56.613870 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.613849 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/707bc0f1-d6de-45c3-9783-dd6772ca19eb-tls-certs\") pod \"model-serving-api-75bd89fdfc-q7657\" (UID: \"707bc0f1-d6de-45c3-9783-dd6772ca19eb\") " pod="opendatahub/model-serving-api-75bd89fdfc-q7657" Nov 03 17:51:56.682212 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.681603 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxtv5\" (UniqueName: \"kubernetes.io/projected/707bc0f1-d6de-45c3-9783-dd6772ca19eb-kube-api-access-rxtv5\") pod \"model-serving-api-75bd89fdfc-q7657\" (UID: \"707bc0f1-d6de-45c3-9783-dd6772ca19eb\") " pod="opendatahub/model-serving-api-75bd89fdfc-q7657" Nov 03 17:51:56.715793 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.715732 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-q7657" Nov 03 17:51:56.976276 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:56.976207 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qtwgk"] Nov 03 17:51:56.981861 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:51:56.981835 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2ca0d7ec_0884_4202_af1f_e49a18e4f0ff.slice/crio-4b347a9f7e28e7a2ef1e1558d26461276de7d368fae099f0393423d9bfc10123 WatchSource:0}: Error finding container 4b347a9f7e28e7a2ef1e1558d26461276de7d368fae099f0393423d9bfc10123: Status 404 returned error can't find the container with id 4b347a9f7e28e7a2ef1e1558d26461276de7d368fae099f0393423d9bfc10123 Nov 03 17:51:57.086342 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:57.086313 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-q7657"] Nov 03 17:51:57.092461 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:51:57.092439 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707bc0f1_d6de_45c3_9783_dd6772ca19eb.slice/crio-0c920b242f19381264285ea2373686672291457f49b1c7ba20bdab3ca8dcb57d WatchSource:0}: Error finding container 0c920b242f19381264285ea2373686672291457f49b1c7ba20bdab3ca8dcb57d: Status 404 returned error can't find the container with id 0c920b242f19381264285ea2373686672291457f49b1c7ba20bdab3ca8dcb57d Nov 03 17:51:57.616267 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:57.616227 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qtwgk" event={"ID":"2ca0d7ec-0884-4202-af1f-e49a18e4f0ff","Type":"ContainerStarted","Data":"4b347a9f7e28e7a2ef1e1558d26461276de7d368fae099f0393423d9bfc10123"} Nov 03 17:51:57.617412 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:57.617370 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-q7657" event={"ID":"707bc0f1-d6de-45c3-9783-dd6772ca19eb","Type":"ContainerStarted","Data":"0c920b242f19381264285ea2373686672291457f49b1c7ba20bdab3ca8dcb57d"} Nov 03 17:51:57.620109 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:57.620089 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:51:59.630453 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:59.630412 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-q7657" event={"ID":"707bc0f1-d6de-45c3-9783-dd6772ca19eb","Type":"ContainerStarted","Data":"6e7fe7f10718e00b6830fd336e55482c06e0467626c23aaba7b24481d93dc6af"} Nov 03 17:51:59.630849 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:59.630477 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-q7657" Nov 03 17:51:59.741694 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:51:59.741598 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-q7657" podStartSLOduration=1.406374226 podStartE2EDuration="3.741579991s" totalImagesPullingTime="2.335205765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:51:59.740038127 +0000 UTC m=+356.873667159" watchObservedRunningTime="2026-11-03 17:51:59.741579991 +0000 UTC m=+356.875208999" Nov 03 17:52:00.611622 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:00.611591 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5" Nov 03 17:52:01.202300 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:01.202255 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:01.409407 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:01.409374 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:01.642924 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:01.642882 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qtwgk" event={"ID":"2ca0d7ec-0884-4202-af1f-e49a18e4f0ff","Type":"ContainerStarted","Data":"f7604316a489bd3f6e5368abe7821a5ab3695cae5d28ec919d30434ce91b8945"} Nov 03 17:52:01.643074 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:01.643019 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qtwgk" Nov 03 17:52:01.753190 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:01.753101 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qtwgk" podStartSLOduration=1.709777528 podStartE2EDuration="5.753086373s" totalImagesPullingTime="4.043308845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:52:01.751525693 +0000 UTC m=+358.885154714" watchObservedRunningTime="2026-11-03 17:52:01.753086373 +0000 UTC m=+358.886715382" Nov 03 17:52:02.911322 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:02.911290 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:03.116520 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:03.116473 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:03.318595 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:03.318548 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:03.519175 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:03.519144 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:03.718291 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:03.718183 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:04.056364 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:04.056337 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:04.265907 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:04.265877 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:06.944490 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:06.944447 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:10.641042 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:10.641011 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-q7657" Nov 03 17:52:11.774951 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:11.774922 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:11.973200 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:11.973172 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:52:14.047945 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.047916 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5"] Nov 03 17:52:14.048320 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.048122 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5" podUID="e948b450-b138-4a11-b189-56d45a657463" containerName="maas-api" containerID="cri-o://5c563de9a2a83ba2011d20f5a43a21df3a0c9492038100eaaa738dea22c892d0" gracePeriod=30 Nov 03 17:52:14.291558 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.291533 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5" Nov 03 17:52:14.368687 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.368660 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e948b450-b138-4a11-b189-56d45a657463-kube-api-access-xvp6d\" (UniqueName: \"kubernetes.io/projected/e948b450-b138-4a11-b189-56d45a657463-kube-api-access-xvp6d\") pod \"e948b450-b138-4a11-b189-56d45a657463\" (UID: \"e948b450-b138-4a11-b189-56d45a657463\") " Nov 03 17:52:14.368855 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.368734 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e948b450-b138-4a11-b189-56d45a657463-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e948b450-b138-4a11-b189-56d45a657463-metrics-certs\") pod \"e948b450-b138-4a11-b189-56d45a657463\" (UID: \"e948b450-b138-4a11-b189-56d45a657463\") " Nov 03 17:52:14.368855 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.368813 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e948b450-b138-4a11-b189-56d45a657463-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e948b450-b138-4a11-b189-56d45a657463-maas-api-tls\") pod \"e948b450-b138-4a11-b189-56d45a657463\" (UID: \"e948b450-b138-4a11-b189-56d45a657463\") " Nov 03 17:52:14.369149 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.368880 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e948b450-b138-4a11-b189-56d45a657463-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e948b450-b138-4a11-b189-56d45a657463-openshift-service-ca\") pod \"e948b450-b138-4a11-b189-56d45a657463\" (UID: \"e948b450-b138-4a11-b189-56d45a657463\") " Nov 03 17:52:14.369223 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.369202 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e948b450-b138-4a11-b189-56d45a657463-openshift-service-ca" pod "e948b450-b138-4a11-b189-56d45a657463" (UID: "e948b450-b138-4a11-b189-56d45a657463"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:52:14.370920 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.370894 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e948b450-b138-4a11-b189-56d45a657463-maas-api-tls" pod "e948b450-b138-4a11-b189-56d45a657463" (UID: "e948b450-b138-4a11-b189-56d45a657463"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:52:14.371181 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.371150 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e948b450-b138-4a11-b189-56d45a657463-metrics-certs" pod "e948b450-b138-4a11-b189-56d45a657463" (UID: "e948b450-b138-4a11-b189-56d45a657463"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:52:14.371181 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.371168 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e948b450-b138-4a11-b189-56d45a657463-kube-api-access-xvp6d" pod "e948b450-b138-4a11-b189-56d45a657463" (UID: "e948b450-b138-4a11-b189-56d45a657463"). InnerVolumeSpecName "kube-api-access-xvp6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:52:14.470421 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.470372 5805 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e948b450-b138-4a11-b189-56d45a657463-metrics-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:14.470421 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.470421 5805 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e948b450-b138-4a11-b189-56d45a657463-maas-api-tls\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:14.470620 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.470433 5805 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e948b450-b138-4a11-b189-56d45a657463-openshift-service-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:14.470620 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.470443 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvp6d\" (UniqueName: \"kubernetes.io/projected/e948b450-b138-4a11-b189-56d45a657463-kube-api-access-xvp6d\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:52:14.703515 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.703431 5805 generic.go:356] "Generic (PLEG): container finished" podID="e948b450-b138-4a11-b189-56d45a657463" containerID="5c563de9a2a83ba2011d20f5a43a21df3a0c9492038100eaaa738dea22c892d0" exitCode=0 Nov 03 17:52:14.703515 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.703495 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5" Nov 03 17:52:14.703714 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.703517 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5" event={"ID":"e948b450-b138-4a11-b189-56d45a657463","Type":"ContainerDied","Data":"5c563de9a2a83ba2011d20f5a43a21df3a0c9492038100eaaa738dea22c892d0"} Nov 03 17:52:14.703714 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.703595 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5" event={"ID":"e948b450-b138-4a11-b189-56d45a657463","Type":"ContainerDied","Data":"d3f623b34c2b867da4a494d6dd4f94e2fa929dc7f07aafc94b5f29590b47e029"} Nov 03 17:52:14.703714 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.703619 5805 scope.go:122] "RemoveContainer" containerID="5c563de9a2a83ba2011d20f5a43a21df3a0c9492038100eaaa738dea22c892d0" Nov 03 17:52:14.713186 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.713172 5805 scope.go:122] "RemoveContainer" containerID="5c563de9a2a83ba2011d20f5a43a21df3a0c9492038100eaaa738dea22c892d0" Nov 03 17:52:14.713459 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:52:14.713439 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c563de9a2a83ba2011d20f5a43a21df3a0c9492038100eaaa738dea22c892d0\": container with ID starting with 5c563de9a2a83ba2011d20f5a43a21df3a0c9492038100eaaa738dea22c892d0 not found: ID does not exist" containerID="5c563de9a2a83ba2011d20f5a43a21df3a0c9492038100eaaa738dea22c892d0" Nov 03 17:52:14.713505 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.713468 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c563de9a2a83ba2011d20f5a43a21df3a0c9492038100eaaa738dea22c892d0"} err="failed to get container status \"5c563de9a2a83ba2011d20f5a43a21df3a0c9492038100eaaa738dea22c892d0\": rpc error: code = NotFound desc = could not find container \"5c563de9a2a83ba2011d20f5a43a21df3a0c9492038100eaaa738dea22c892d0\": container with ID starting with 5c563de9a2a83ba2011d20f5a43a21df3a0c9492038100eaaa738dea22c892d0 not found: ID does not exist" Nov 03 17:52:14.857646 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.857619 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5"] Nov 03 17:52:14.903622 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.903596 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-vzfx5"] Nov 03 17:52:14.975019 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:14.974958 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e948b450-b138-4a11-b189-56d45a657463" path="/var/lib/kubelet/pods/e948b450-b138-4a11-b189-56d45a657463/volumes" Nov 03 17:52:26.233866 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:52:26.233822 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:52:32.650307 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:52:32.650264 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qtwgk" Nov 03 17:52:56.233740 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:52:56.233700 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:53:05.980857 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:05.980821 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-zwb45"] Nov 03 17:53:05.981244 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:05.981205 5805 cpu_manager.go:414] "containerMap: removing container" podUID="e948b450-b138-4a11-b189-56d45a657463" containerName="maas-api" Nov 03 17:53:05.981244 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:05.981216 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e948b450-b138-4a11-b189-56d45a657463" containerName="maas-api" Nov 03 17:53:05.981313 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:05.981301 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="e948b450-b138-4a11-b189-56d45a657463" containerName="maas-api" Nov 03 17:53:05.984549 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:05.984524 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-zwb45" Nov 03 17:53:06.019958 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:06.019931 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c3ba816-8be5-4c56-8be8-6f7dae15d058-cert\") pod \"llmisvc-controller-manager-6466c554d8-zwb45\" (UID: \"7c3ba816-8be5-4c56-8be8-6f7dae15d058\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-zwb45" Nov 03 17:53:06.020078 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:06.019986 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6w8b\" (UniqueName: \"kubernetes.io/projected/7c3ba816-8be5-4c56-8be8-6f7dae15d058-kube-api-access-m6w8b\") pod \"llmisvc-controller-manager-6466c554d8-zwb45\" (UID: \"7c3ba816-8be5-4c56-8be8-6f7dae15d058\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-zwb45" Nov 03 17:53:06.041446 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:06.041420 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-zwb45"] Nov 03 17:53:06.120811 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:06.120783 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c3ba816-8be5-4c56-8be8-6f7dae15d058-cert\") pod \"llmisvc-controller-manager-6466c554d8-zwb45\" (UID: \"7c3ba816-8be5-4c56-8be8-6f7dae15d058\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-zwb45" Nov 03 17:53:06.120994 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:06.120825 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6w8b\" (UniqueName: \"kubernetes.io/projected/7c3ba816-8be5-4c56-8be8-6f7dae15d058-kube-api-access-m6w8b\") pod \"llmisvc-controller-manager-6466c554d8-zwb45\" (UID: \"7c3ba816-8be5-4c56-8be8-6f7dae15d058\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-zwb45" Nov 03 17:53:06.123359 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:06.123331 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c3ba816-8be5-4c56-8be8-6f7dae15d058-cert\") pod \"llmisvc-controller-manager-6466c554d8-zwb45\" (UID: \"7c3ba816-8be5-4c56-8be8-6f7dae15d058\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-zwb45" Nov 03 17:53:06.178471 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:06.178441 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6w8b\" (UniqueName: \"kubernetes.io/projected/7c3ba816-8be5-4c56-8be8-6f7dae15d058-kube-api-access-m6w8b\") pod \"llmisvc-controller-manager-6466c554d8-zwb45\" (UID: \"7c3ba816-8be5-4c56-8be8-6f7dae15d058\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-zwb45" Nov 03 17:53:06.306565 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:06.306539 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-zwb45" Nov 03 17:53:06.675889 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:06.675799 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-zwb45"] Nov 03 17:53:06.682429 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:53:06.682385 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3ba816_8be5_4c56_8be8_6f7dae15d058.slice/crio-db251623c3017567c61dcafa947688425a387c032d3978f576d3aad3ef44c4d2 WatchSource:0}: Error finding container db251623c3017567c61dcafa947688425a387c032d3978f576d3aad3ef44c4d2: Status 404 returned error can't find the container with id db251623c3017567c61dcafa947688425a387c032d3978f576d3aad3ef44c4d2 Nov 03 17:53:06.954894 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:06.954814 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-zwb45" event={"ID":"7c3ba816-8be5-4c56-8be8-6f7dae15d058","Type":"ContainerStarted","Data":"db251623c3017567c61dcafa947688425a387c032d3978f576d3aad3ef44c4d2"} Nov 03 17:53:07.311681 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:07.311649 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:07.961143 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:07.961062 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-zwb45" event={"ID":"7c3ba816-8be5-4c56-8be8-6f7dae15d058","Type":"ContainerStarted","Data":"3fadf84b360f20665081a41dcf43a2ac09ed02bed8d09d2346746441b9be081f"} Nov 03 17:53:07.961281 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:07.961209 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-zwb45" Nov 03 17:53:08.083432 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:08.083362 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-zwb45" podStartSLOduration=2.222151471 podStartE2EDuration="3.083348307s" totalImagesPullingTime="861.196836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:53:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:53:08.08299844 +0000 UTC m=+425.216627454" watchObservedRunningTime="2026-11-03 17:53:08.083348307 +0000 UTC m=+425.216977317" Nov 03 17:53:21.274058 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:21.274015 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:21.478375 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:21.478339 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:26.233285 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:53:26.233243 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:53:38.974085 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:38.974056 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-zwb45" Nov 03 17:53:39.131963 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:39.131924 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qtwgk"] Nov 03 17:53:39.132185 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:39.132155 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qtwgk" podUID="2ca0d7ec-0884-4202-af1f-e49a18e4f0ff" containerName="manager" containerID="cri-o://f7604316a489bd3f6e5368abe7821a5ab3695cae5d28ec919d30434ce91b8945" gracePeriod=30 Nov 03 17:53:39.371185 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:39.371157 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qtwgk" Nov 03 17:53:39.504916 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:39.504806 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ca0d7ec-0884-4202-af1f-e49a18e4f0ff-cert\" (UniqueName: \"kubernetes.io/secret/2ca0d7ec-0884-4202-af1f-e49a18e4f0ff-cert\") pod \"2ca0d7ec-0884-4202-af1f-e49a18e4f0ff\" (UID: \"2ca0d7ec-0884-4202-af1f-e49a18e4f0ff\") " Nov 03 17:53:39.504916 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:39.504878 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ca0d7ec-0884-4202-af1f-e49a18e4f0ff-kube-api-access-sr86k\" (UniqueName: \"kubernetes.io/projected/2ca0d7ec-0884-4202-af1f-e49a18e4f0ff-kube-api-access-sr86k\") pod \"2ca0d7ec-0884-4202-af1f-e49a18e4f0ff\" (UID: \"2ca0d7ec-0884-4202-af1f-e49a18e4f0ff\") " Nov 03 17:53:39.506948 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:39.506914 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca0d7ec-0884-4202-af1f-e49a18e4f0ff-cert" pod "2ca0d7ec-0884-4202-af1f-e49a18e4f0ff" (UID: "2ca0d7ec-0884-4202-af1f-e49a18e4f0ff"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:53:39.506948 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:39.506936 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ca0d7ec-0884-4202-af1f-e49a18e4f0ff-kube-api-access-sr86k" pod "2ca0d7ec-0884-4202-af1f-e49a18e4f0ff" (UID: "2ca0d7ec-0884-4202-af1f-e49a18e4f0ff"). InnerVolumeSpecName "kube-api-access-sr86k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:53:39.606448 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:39.606423 5805 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ca0d7ec-0884-4202-af1f-e49a18e4f0ff-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:39.606448 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:39.606446 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr86k\" (UniqueName: \"kubernetes.io/projected/2ca0d7ec-0884-4202-af1f-e49a18e4f0ff-kube-api-access-sr86k\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 17:53:40.036943 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:40.036895 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:40.112314 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:40.112279 5805 generic.go:356] "Generic (PLEG): container finished" podID="2ca0d7ec-0884-4202-af1f-e49a18e4f0ff" containerID="f7604316a489bd3f6e5368abe7821a5ab3695cae5d28ec919d30434ce91b8945" exitCode=0 Nov 03 17:53:40.112314 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:40.112317 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qtwgk" Nov 03 17:53:40.112528 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:40.112361 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qtwgk" event={"ID":"2ca0d7ec-0884-4202-af1f-e49a18e4f0ff","Type":"ContainerDied","Data":"f7604316a489bd3f6e5368abe7821a5ab3695cae5d28ec919d30434ce91b8945"} Nov 03 17:53:40.112528 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:40.112428 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-qtwgk" event={"ID":"2ca0d7ec-0884-4202-af1f-e49a18e4f0ff","Type":"ContainerDied","Data":"4b347a9f7e28e7a2ef1e1558d26461276de7d368fae099f0393423d9bfc10123"} Nov 03 17:53:40.112528 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:40.112453 5805 scope.go:122] "RemoveContainer" containerID="f7604316a489bd3f6e5368abe7821a5ab3695cae5d28ec919d30434ce91b8945" Nov 03 17:53:40.124555 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:40.124538 5805 scope.go:122] "RemoveContainer" containerID="f7604316a489bd3f6e5368abe7821a5ab3695cae5d28ec919d30434ce91b8945" Nov 03 17:53:40.124799 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:53:40.124781 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7604316a489bd3f6e5368abe7821a5ab3695cae5d28ec919d30434ce91b8945\": container with ID starting with f7604316a489bd3f6e5368abe7821a5ab3695cae5d28ec919d30434ce91b8945 not found: ID does not exist" containerID="f7604316a489bd3f6e5368abe7821a5ab3695cae5d28ec919d30434ce91b8945" Nov 03 17:53:40.124851 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:40.124808 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7604316a489bd3f6e5368abe7821a5ab3695cae5d28ec919d30434ce91b8945"} err="failed to get container status \"f7604316a489bd3f6e5368abe7821a5ab3695cae5d28ec919d30434ce91b8945\": rpc error: code = NotFound desc = could not find container \"f7604316a489bd3f6e5368abe7821a5ab3695cae5d28ec919d30434ce91b8945\": container with ID starting with f7604316a489bd3f6e5368abe7821a5ab3695cae5d28ec919d30434ce91b8945 not found: ID does not exist" Nov 03 17:53:40.241877 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:40.241849 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:40.268152 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:40.268128 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qtwgk"] Nov 03 17:53:40.314153 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:40.314130 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-qtwgk"] Nov 03 17:53:40.975594 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:40.975565 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ca0d7ec-0884-4202-af1f-e49a18e4f0ff" path="/var/lib/kubelet/pods/2ca0d7ec-0884-4202-af1f-e49a18e4f0ff/volumes" Nov 03 17:53:56.235421 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:53:56.235316 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:53:57.386460 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:57.386417 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:53:57.589539 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:53:57.589509 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:01.510482 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:01.510449 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:04.830540 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:04.830501 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:07.648113 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:07.648083 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:07.969101 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:07.969021 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:12.689666 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:12.689632 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:15.871797 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:15.871767 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:18.907043 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:18.907013 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:24.908761 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:24.908717 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:26.233557 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:54:26.233521 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:54:30.184281 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:30.184246 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:33.831739 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:33.831709 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph"] Nov 03 17:54:33.832115 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:33.832096 5805 cpu_manager.go:414] "containerMap: removing container" podUID="2ca0d7ec-0884-4202-af1f-e49a18e4f0ff" containerName="manager" Nov 03 17:54:33.832115 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:33.832107 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ca0d7ec-0884-4202-af1f-e49a18e4f0ff" containerName="manager" Nov 03 17:54:33.832187 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:33.832167 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ca0d7ec-0884-4202-af1f-e49a18e4f0ff" containerName="manager" Nov 03 17:54:33.835319 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:33.835293 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:33.884428 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:33.884384 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h54vm\"" Nov 03 17:54:33.932248 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:33.932224 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph"] Nov 03 17:54:33.982194 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:33.982168 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d685b092-b52b-4ca9-a114-dd71a3be6f40-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:33.982326 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:33.982200 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d685b092-b52b-4ca9-a114-dd71a3be6f40-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:33.982376 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:33.982325 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d685b092-b52b-4ca9-a114-dd71a3be6f40-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:33.982376 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:33.982367 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d685b092-b52b-4ca9-a114-dd71a3be6f40-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:33.982489 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:33.982390 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mhxc\" (UniqueName: \"kubernetes.io/projected/d685b092-b52b-4ca9-a114-dd71a3be6f40-kube-api-access-5mhxc\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:33.982489 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:33.982469 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d685b092-b52b-4ca9-a114-dd71a3be6f40-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:33.982570 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:33.982490 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d685b092-b52b-4ca9-a114-dd71a3be6f40-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:33.982570 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:33.982508 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d685b092-b52b-4ca9-a114-dd71a3be6f40-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:33.982570 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:33.982552 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d685b092-b52b-4ca9-a114-dd71a3be6f40-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:34.083981 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:34.083914 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d685b092-b52b-4ca9-a114-dd71a3be6f40-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:34.083981 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:34.083953 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d685b092-b52b-4ca9-a114-dd71a3be6f40-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:34.083981 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:34.083973 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mhxc\" (UniqueName: \"kubernetes.io/projected/d685b092-b52b-4ca9-a114-dd71a3be6f40-kube-api-access-5mhxc\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:34.084172 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:34.084091 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d685b092-b52b-4ca9-a114-dd71a3be6f40-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:34.084172 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:34.084125 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d685b092-b52b-4ca9-a114-dd71a3be6f40-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:34.084172 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:34.084152 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d685b092-b52b-4ca9-a114-dd71a3be6f40-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:34.084318 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:34.084175 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d685b092-b52b-4ca9-a114-dd71a3be6f40-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:34.084318 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:34.084238 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d685b092-b52b-4ca9-a114-dd71a3be6f40-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:34.084318 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:34.084265 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d685b092-b52b-4ca9-a114-dd71a3be6f40-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:34.084512 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:34.084359 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d685b092-b52b-4ca9-a114-dd71a3be6f40-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:34.084576 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:34.084534 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d685b092-b52b-4ca9-a114-dd71a3be6f40-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:34.084649 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:34.084626 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d685b092-b52b-4ca9-a114-dd71a3be6f40-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:34.084704 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:34.084660 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d685b092-b52b-4ca9-a114-dd71a3be6f40-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:34.084809 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:34.084791 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d685b092-b52b-4ca9-a114-dd71a3be6f40-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:34.086265 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:34.086243 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d685b092-b52b-4ca9-a114-dd71a3be6f40-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:34.086512 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:34.086493 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d685b092-b52b-4ca9-a114-dd71a3be6f40-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:34.143936 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:34.143909 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mhxc\" (UniqueName: \"kubernetes.io/projected/d685b092-b52b-4ca9-a114-dd71a3be6f40-kube-api-access-5mhxc\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:34.143936 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:34.143918 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d685b092-b52b-4ca9-a114-dd71a3be6f40-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-vgzph\" (UID: \"d685b092-b52b-4ca9-a114-dd71a3be6f40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:34.147763 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:34.147746 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:34.571136 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:34.571107 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph"] Nov 03 17:54:34.576766 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:54:34.576741 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd685b092_b52b_4ca9_a114_dd71a3be6f40.slice/crio-d61d4c0c04e7cefa09f2ed4d6802d4b6029a1f9ab7fb25afb0228fe21ed18ef6 WatchSource:0}: Error finding container d61d4c0c04e7cefa09f2ed4d6802d4b6029a1f9ab7fb25afb0228fe21ed18ef6: Status 404 returned error can't find the container with id d61d4c0c04e7cefa09f2ed4d6802d4b6029a1f9ab7fb25afb0228fe21ed18ef6 Nov 03 17:54:34.578839 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:34.578809 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:54:34.578899 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:34.578877 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:54:34.578933 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:34.578904 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:54:35.382475 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:35.382431 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" event={"ID":"d685b092-b52b-4ca9-a114-dd71a3be6f40","Type":"ContainerStarted","Data":"459639033b3d810a1f4c0cd76dfa3544d6588fa3e62af2bb8a10db0e1cd9a59c"} Nov 03 17:54:35.382847 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:35.382509 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" event={"ID":"d685b092-b52b-4ca9-a114-dd71a3be6f40","Type":"ContainerStarted","Data":"d61d4c0c04e7cefa09f2ed4d6802d4b6029a1f9ab7fb25afb0228fe21ed18ef6"} Nov 03 17:54:35.499970 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:35.499921 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" podStartSLOduration=2.499903835 podStartE2EDuration="2.499903835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:54:35.498145514 +0000 UTC m=+512.631774525" watchObservedRunningTime="2026-11-03 17:54:35.499903835 +0000 UTC m=+512.633532847" Nov 03 17:54:36.148742 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:36.148700 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:36.149872 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:36.149844 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vgzph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 17:54:36.149964 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:36.149907 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" podUID="d685b092-b52b-4ca9-a114-dd71a3be6f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 17:54:37.148224 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:37.148194 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vgzph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 17:54:37.148611 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:37.148245 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" podUID="d685b092-b52b-4ca9-a114-dd71a3be6f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 17:54:38.148357 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:38.148324 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vgzph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 17:54:38.148789 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:38.148385 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" podUID="d685b092-b52b-4ca9-a114-dd71a3be6f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 17:54:39.149075 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:39.149034 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vgzph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 17:54:39.149468 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:39.149103 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" podUID="d685b092-b52b-4ca9-a114-dd71a3be6f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 17:54:40.148269 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:40.148234 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vgzph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 17:54:40.148568 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:40.148532 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" podUID="d685b092-b52b-4ca9-a114-dd71a3be6f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 17:54:41.148676 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:41.148629 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vgzph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 17:54:41.149176 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:41.148696 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" podUID="d685b092-b52b-4ca9-a114-dd71a3be6f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 17:54:43.148616 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:43.148572 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vgzph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:54:43.149028 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:43.148680 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" podUID="d685b092-b52b-4ca9-a114-dd71a3be6f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:54:44.190553 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:44.190508 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vgzph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:54:44.190961 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:44.190582 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" podUID="d685b092-b52b-4ca9-a114-dd71a3be6f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:54:44.479115 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:44.479039 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:44.479456 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:44.479436 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:44.480009 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:44.479987 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" Nov 03 17:54:56.234512 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:54:56.234476 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:54:56.321490 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:56.321462 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:54:57.948701 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:54:57.948660 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:55:05.385889 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:05.385851 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:55:05.587344 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:05.587310 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:55:08.022208 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:08.022166 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:55:09.046850 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:09.046818 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:55:09.255061 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:09.255028 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:55:19.184763 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.184687 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644"] Nov 03 17:55:19.188265 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.188244 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.285330 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.285298 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644"] Nov 03 17:55:19.317564 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.317534 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.317723 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.317578 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.317723 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.317603 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.317847 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.317768 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.317847 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.317813 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.317947 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.317929 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.317983 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.317958 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.317983 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.317976 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6jpf\" (UniqueName: \"kubernetes.io/projected/293f67c8-c8cc-4198-8e03-79c4ae3a710c-kube-api-access-b6jpf\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.318054 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.318013 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.418699 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.418673 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.418834 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.418709 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.418834 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.418728 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.418909 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.418854 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.418909 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.418897 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.418998 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.418980 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.419050 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.419017 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.419050 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.419041 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6jpf\" (UniqueName: \"kubernetes.io/projected/293f67c8-c8cc-4198-8e03-79c4ae3a710c-kube-api-access-b6jpf\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.419140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.419077 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.419140 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.419109 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.419230 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.419219 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.419287 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.419259 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.419482 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.419460 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.419792 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.419775 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.421128 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.421108 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.421220 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.421157 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.477923 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.477861 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6jpf\" (UniqueName: \"kubernetes.io/projected/293f67c8-c8cc-4198-8e03-79c4ae3a710c-kube-api-access-b6jpf\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.477923 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.477912 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-hn644\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.501528 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.501505 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:19.931252 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.931216 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644"] Nov 03 17:55:19.937763 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:55:19.937731 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod293f67c8_c8cc_4198_8e03_79c4ae3a710c.slice/crio-f369560ad0064258c2d1fa14751b2e3b3d9b96595ab86d4352111826b72fa7db WatchSource:0}: Error finding container f369560ad0064258c2d1fa14751b2e3b3d9b96595ab86d4352111826b72fa7db: Status 404 returned error can't find the container with id f369560ad0064258c2d1fa14751b2e3b3d9b96595ab86d4352111826b72fa7db Nov 03 17:55:19.939853 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.939823 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:55:19.939940 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.939890 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:55:19.939940 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:19.939919 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:55:20.601700 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:20.601659 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" event={"ID":"293f67c8-c8cc-4198-8e03-79c4ae3a710c","Type":"ContainerStarted","Data":"9c63ffcc383c8a7996bde478d980c6bb632293d4596dfd5a6272d164b901d810"} Nov 03 17:55:20.602083 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:20.601719 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" event={"ID":"293f67c8-c8cc-4198-8e03-79c4ae3a710c","Type":"ContainerStarted","Data":"f369560ad0064258c2d1fa14751b2e3b3d9b96595ab86d4352111826b72fa7db"} Nov 03 17:55:20.717153 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:20.717104 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" podStartSLOduration=1.717089493 podStartE2EDuration="1.717089493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:55:20.715307394 +0000 UTC m=+557.848936419" watchObservedRunningTime="2026-11-03 17:55:20.717089493 +0000 UTC m=+557.850718503" Nov 03 17:55:21.502191 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:21.502154 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:21.503445 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:21.503419 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hn644 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 17:55:21.503567 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:21.503465 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" podUID="293f67c8-c8cc-4198-8e03-79c4ae3a710c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 17:55:22.501954 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:22.501916 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hn644 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 17:55:22.502365 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:22.501974 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" podUID="293f67c8-c8cc-4198-8e03-79c4ae3a710c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 17:55:23.501764 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:23.501724 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hn644 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 17:55:23.502064 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:23.501781 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" podUID="293f67c8-c8cc-4198-8e03-79c4ae3a710c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 17:55:24.501908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:24.501870 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hn644 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 17:55:24.502125 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:24.501935 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" podUID="293f67c8-c8cc-4198-8e03-79c4ae3a710c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 17:55:25.502889 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:25.502852 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hn644 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 17:55:25.503326 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:25.502911 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" podUID="293f67c8-c8cc-4198-8e03-79c4ae3a710c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 17:55:26.234904 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:55:26.234842 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:55:26.502028 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:26.501948 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-hn644 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 17:55:26.502028 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:26.501998 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" podUID="293f67c8-c8cc-4198-8e03-79c4ae3a710c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 17:55:28.039793 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:28.039757 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:28.040339 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:28.040122 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:28.041017 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:28.040997 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 17:55:35.125579 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:35.125547 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:55:35.654781 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:35.654742 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:55:37.311283 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:37.311251 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:55:37.570332 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:37.570249 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:55:39.108587 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:39.108554 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:55:39.320965 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:39.320928 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:55:42.908644 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:42.908613 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:55:55.354264 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:55.354228 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:55:55.672864 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:55:55.672789 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:55:56.234832 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:55:56.234799 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:56:03.061525 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:03.061492 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 17:56:03.062848 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:03.062825 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 17:56:03.955284 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:03.955241 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:06.394659 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:06.394596 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:06.602533 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:06.602501 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:07.129598 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:07.129552 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:07.335755 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:07.335715 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:07.555643 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:07.555558 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:07.757363 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:07.757331 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:07.998735 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:07.998697 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:08.222265 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:08.222229 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:08.529752 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:08.529706 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:08.829835 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:08.829797 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:09.161650 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:09.161553 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:09.369994 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:09.369965 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:11.591417 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:11.591368 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:11.805557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:11.805525 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:12.181358 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:12.181323 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:12.432692 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:12.432604 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:12.675166 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:12.675131 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:13.006957 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:13.006921 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:13.222183 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:13.222134 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:13.431990 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:13.431945 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:13.805756 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:13.805686 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:14.077904 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:14.077875 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:14.401860 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:14.401766 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:14.609525 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:14.609490 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:16.576548 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:16.576511 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:16.779183 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:16.779143 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:17.024102 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:17.024021 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:17.288027 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:17.287946 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:17.520509 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:17.520478 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:17.899149 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:17.899114 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:18.103330 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:18.103262 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:18.721877 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:18.721838 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:19.034830 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:19.034741 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:21.724150 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:21.724116 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:21.924790 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:21.924752 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:22.931931 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:22.931895 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:23.137583 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:23.137548 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:23.333241 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:23.333209 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:24.225265 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:24.225230 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:24.426505 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:24.426473 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:24.694844 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:24.694813 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg"] Nov 03 17:56:24.698328 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:24.698309 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg" Nov 03 17:56:24.793242 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:24.793203 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-aab753\"" Nov 03 17:56:24.793445 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:24.793250 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-aab753-dockercfg-54xmm\"" Nov 03 17:56:24.795211 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:24.795185 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg"] Nov 03 17:56:24.816794 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:24.816761 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9n7m\" (UniqueName: \"kubernetes.io/projected/7195e16b-0603-4421-92be-3cff9a0fa7eb-kube-api-access-d9n7m\") pod \"payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg\" (UID: \"7195e16b-0603-4421-92be-3cff9a0fa7eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg" Nov 03 17:56:24.816928 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:24.816878 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7195e16b-0603-4421-92be-3cff9a0fa7eb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg\" (UID: \"7195e16b-0603-4421-92be-3cff9a0fa7eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg" Nov 03 17:56:24.917556 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:24.917520 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7195e16b-0603-4421-92be-3cff9a0fa7eb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg\" (UID: \"7195e16b-0603-4421-92be-3cff9a0fa7eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg" Nov 03 17:56:24.917732 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:24.917607 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9n7m\" (UniqueName: \"kubernetes.io/projected/7195e16b-0603-4421-92be-3cff9a0fa7eb-kube-api-access-d9n7m\") pod \"payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg\" (UID: \"7195e16b-0603-4421-92be-3cff9a0fa7eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg" Nov 03 17:56:24.918202 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:24.918182 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7195e16b-0603-4421-92be-3cff9a0fa7eb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg\" (UID: \"7195e16b-0603-4421-92be-3cff9a0fa7eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg" Nov 03 17:56:24.975211 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:24.975139 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9n7m\" (UniqueName: \"kubernetes.io/projected/7195e16b-0603-4421-92be-3cff9a0fa7eb-kube-api-access-d9n7m\") pod \"payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg\" (UID: \"7195e16b-0603-4421-92be-3cff9a0fa7eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg" Nov 03 17:56:25.012040 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:25.012011 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg" Nov 03 17:56:25.380973 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:25.380937 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg"] Nov 03 17:56:25.387595 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:56:25.387567 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7195e16b_0603_4421_92be_3cff9a0fa7eb.slice/crio-6f83557bd8502bbcb33a06db49e0a4cb134f3e362eac2c44331d6c8c01f73ec0 WatchSource:0}: Error finding container 6f83557bd8502bbcb33a06db49e0a4cb134f3e362eac2c44331d6c8c01f73ec0: Status 404 returned error can't find the container with id 6f83557bd8502bbcb33a06db49e0a4cb134f3e362eac2c44331d6c8c01f73ec0 Nov 03 17:56:25.389500 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:25.389482 5805 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 17:56:25.948092 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:25.948058 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg" event={"ID":"7195e16b-0603-4421-92be-3cff9a0fa7eb","Type":"ContainerStarted","Data":"6f83557bd8502bbcb33a06db49e0a4cb134f3e362eac2c44331d6c8c01f73ec0"} Nov 03 17:56:26.235523 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:56:26.235437 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:56:26.757091 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:26.757043 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:26.960877 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:26.960841 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:27.095279 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.095243 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9"] Nov 03 17:56:27.099029 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.099005 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9" Nov 03 17:56:27.138231 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.138203 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfkjk\" (UniqueName: \"kubernetes.io/projected/d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5-kube-api-access-vfkjk\") pod \"payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9\" (UID: \"d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9" Nov 03 17:56:27.138428 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.138253 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9\" (UID: \"d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9" Nov 03 17:56:27.148106 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.148080 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-2b4f46\"" Nov 03 17:56:27.197958 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.197923 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-2b4f46-dockercfg-zk2fk\"" Nov 03 17:56:27.199591 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.199563 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9"] Nov 03 17:56:27.239216 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.239172 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfkjk\" (UniqueName: \"kubernetes.io/projected/d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5-kube-api-access-vfkjk\") pod \"payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9\" (UID: \"d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9" Nov 03 17:56:27.239411 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.239223 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9\" (UID: \"d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9" Nov 03 17:56:27.239890 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.239869 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9\" (UID: \"d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9" Nov 03 17:56:27.262482 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.262452 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w"] Nov 03 17:56:27.268088 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.268059 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w" Nov 03 17:56:27.296625 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.296595 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfkjk\" (UniqueName: \"kubernetes.io/projected/d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5-kube-api-access-vfkjk\") pod \"payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9\" (UID: \"d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9" Nov 03 17:56:27.322042 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.322011 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w"] Nov 03 17:56:27.340487 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.340449 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkdj2\" (UniqueName: \"kubernetes.io/projected/d41d4083-b660-4c12-8b03-47159d59b941-kube-api-access-nkdj2\") pod \"payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w\" (UID: \"d41d4083-b660-4c12-8b03-47159d59b941\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w" Nov 03 17:56:27.340834 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.340623 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d41d4083-b660-4c12-8b03-47159d59b941-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w\" (UID: \"d41d4083-b660-4c12-8b03-47159d59b941\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w" Nov 03 17:56:27.414429 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.414329 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9" Nov 03 17:56:27.441965 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.441932 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkdj2\" (UniqueName: \"kubernetes.io/projected/d41d4083-b660-4c12-8b03-47159d59b941-kube-api-access-nkdj2\") pod \"payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w\" (UID: \"d41d4083-b660-4c12-8b03-47159d59b941\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w" Nov 03 17:56:27.442098 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.442073 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d41d4083-b660-4c12-8b03-47159d59b941-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w\" (UID: \"d41d4083-b660-4c12-8b03-47159d59b941\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w" Nov 03 17:56:27.443173 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.443146 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d41d4083-b660-4c12-8b03-47159d59b941-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w\" (UID: \"d41d4083-b660-4c12-8b03-47159d59b941\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w" Nov 03 17:56:27.505965 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.505648 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkdj2\" (UniqueName: \"kubernetes.io/projected/d41d4083-b660-4c12-8b03-47159d59b941-kube-api-access-nkdj2\") pod \"payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w\" (UID: \"d41d4083-b660-4c12-8b03-47159d59b941\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w" Nov 03 17:56:27.584676 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.584638 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w" Nov 03 17:56:27.803643 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.803575 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9"] Nov 03 17:56:27.809983 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:56:27.809955 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5cd7c49_cbc5_404d_bfb2_ef58348f7ae5.slice/crio-e02bb38b8b924a2d688915b14de9ee32cd978b772cbc2a100032cd472e695472 WatchSource:0}: Error finding container e02bb38b8b924a2d688915b14de9ee32cd978b772cbc2a100032cd472e695472: Status 404 returned error can't find the container with id e02bb38b8b924a2d688915b14de9ee32cd978b772cbc2a100032cd472e695472 Nov 03 17:56:27.959128 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.959051 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9" event={"ID":"d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5","Type":"ContainerStarted","Data":"e02bb38b8b924a2d688915b14de9ee32cd978b772cbc2a100032cd472e695472"} Nov 03 17:56:27.962038 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.962008 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w"] Nov 03 17:56:27.966806 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:56:27.966775 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd41d4083_b660_4c12_8b03_47159d59b941.slice/crio-2b6e46332b24e055a273edf0b428e7147231a4bdb3a38ac04b3ecf56d318f36e WatchSource:0}: Error finding container 2b6e46332b24e055a273edf0b428e7147231a4bdb3a38ac04b3ecf56d318f36e: Status 404 returned error can't find the container with id 2b6e46332b24e055a273edf0b428e7147231a4bdb3a38ac04b3ecf56d318f36e Nov 03 17:56:27.973757 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:27.973727 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:28.178068 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:28.178028 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:28.965330 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:28.965237 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w" event={"ID":"d41d4083-b660-4c12-8b03-47159d59b941","Type":"ContainerStarted","Data":"8d0bd81b01f51fd2fd02559dd33e87cde0e6d7e0a9bda422976d4715f46febb4"} Nov 03 17:56:28.965330 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:28.965296 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w" event={"ID":"d41d4083-b660-4c12-8b03-47159d59b941","Type":"ContainerStarted","Data":"2b6e46332b24e055a273edf0b428e7147231a4bdb3a38ac04b3ecf56d318f36e"} Nov 03 17:56:28.965330 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:28.965323 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w" Nov 03 17:56:28.966925 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:28.966899 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9" event={"ID":"d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5","Type":"ContainerStarted","Data":"6e024dc9c05442738aa29126f0fb09f3bc93954387acc37a8b56878fea1e7877"} Nov 03 17:56:28.967064 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:28.966950 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9" Nov 03 17:56:28.973699 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:28.973681 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg" Nov 03 17:56:28.973796 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:28.973721 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg" event={"ID":"7195e16b-0603-4421-92be-3cff9a0fa7eb","Type":"ContainerStarted","Data":"a695737ae7791cfd82401e824525e5204470e0e736496fe78d02bbca854f76db"} Nov 03 17:56:29.079215 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:29.079151 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w" podStartSLOduration=1.2382379829999999 podStartE2EDuration="2.079133012s" totalImagesPullingTime="840.895029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:29.078641007 +0000 UTC m=+626.212270029" watchObservedRunningTime="2026-11-03 17:56:29.079133012 +0000 UTC m=+626.212762024" Nov 03 17:56:29.187656 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:29.187596 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg" podStartSLOduration=2.400333008 podStartE2EDuration="5.187579274s" totalImagesPullingTime="2.787246266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:29.187339408 +0000 UTC m=+626.320968430" watchObservedRunningTime="2026-11-03 17:56:29.187579274 +0000 UTC m=+626.321208285" Nov 03 17:56:29.299705 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:29.299606 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9" podStartSLOduration=1.436131194 podStartE2EDuration="2.299590384s" totalImagesPullingTime="863.45919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:29.297608291 +0000 UTC m=+626.431237336" watchObservedRunningTime="2026-11-03 17:56:29.299590384 +0000 UTC m=+626.433219391" Nov 03 17:56:31.710209 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:31.710173 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:31.924506 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:31.924467 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:32.779697 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:32.779669 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42"] Nov 03 17:56:32.783470 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:32.783451 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" Nov 03 17:56:32.832298 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:32.832278 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-2b4f46-serving-cert\"" Nov 03 17:56:32.878732 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:32.878680 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tr54c\"" Nov 03 17:56:32.878870 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:32.878760 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 17:56:32.880999 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:32.880978 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42"] Nov 03 17:56:32.897172 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:32.897146 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42\" (UID: \"c5e3b992-04fc-4c47-b353-55a0fa75f7ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" Nov 03 17:56:32.897300 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:32.897200 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49tkb\" (UniqueName: \"kubernetes.io/projected/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-kube-api-access-49tkb\") pod \"maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42\" (UID: \"c5e3b992-04fc-4c47-b353-55a0fa75f7ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" Nov 03 17:56:32.897370 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:32.897302 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-metrics-certs\") pod \"maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42\" (UID: \"c5e3b992-04fc-4c47-b353-55a0fa75f7ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" Nov 03 17:56:32.897370 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:32.897357 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-maas-api-tls\") pod \"maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42\" (UID: \"c5e3b992-04fc-4c47-b353-55a0fa75f7ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" Nov 03 17:56:32.998718 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:32.998688 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42\" (UID: \"c5e3b992-04fc-4c47-b353-55a0fa75f7ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" Nov 03 17:56:32.998907 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:32.998743 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49tkb\" (UniqueName: \"kubernetes.io/projected/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-kube-api-access-49tkb\") pod \"maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42\" (UID: \"c5e3b992-04fc-4c47-b353-55a0fa75f7ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" Nov 03 17:56:32.998907 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:32.998770 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-metrics-certs\") pod \"maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42\" (UID: \"c5e3b992-04fc-4c47-b353-55a0fa75f7ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" Nov 03 17:56:32.998907 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:32.998797 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-maas-api-tls\") pod \"maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42\" (UID: \"c5e3b992-04fc-4c47-b353-55a0fa75f7ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" Nov 03 17:56:32.998907 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:56:32.998907 5805 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-serving-cert: secret "maas-api-e2e-worker-w5-2b4f46-serving-cert" not found Nov 03 17:56:32.999155 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:56:32.998964 5805 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-maas-api-tls podName:c5e3b992-04fc-4c47-b353-55a0fa75f7ed nodeName:}" failed. No retries permitted until 2026-11-03 17:56:33.498950301 +0000 UTC m=+630.632579302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-maas-api-tls") pod "maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" (UID: "c5e3b992-04fc-4c47-b353-55a0fa75f7ed") : secret "maas-api-e2e-worker-w5-2b4f46-serving-cert" not found Nov 03 17:56:32.999475 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:32.999450 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42\" (UID: \"c5e3b992-04fc-4c47-b353-55a0fa75f7ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" Nov 03 17:56:33.001117 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:33.001096 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-metrics-certs\") pod \"maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42\" (UID: \"c5e3b992-04fc-4c47-b353-55a0fa75f7ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" Nov 03 17:56:33.056933 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:33.056903 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49tkb\" (UniqueName: \"kubernetes.io/projected/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-kube-api-access-49tkb\") pod \"maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42\" (UID: \"c5e3b992-04fc-4c47-b353-55a0fa75f7ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" Nov 03 17:56:33.061862 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:33.061820 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:33.254118 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:33.254063 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:33.503644 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:33.503563 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-maas-api-tls\") pod \"maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42\" (UID: \"c5e3b992-04fc-4c47-b353-55a0fa75f7ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" Nov 03 17:56:33.505838 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:33.505811 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-maas-api-tls\") pod \"maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42\" (UID: \"c5e3b992-04fc-4c47-b353-55a0fa75f7ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" Nov 03 17:56:33.697124 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:33.697090 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" Nov 03 17:56:34.073774 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:34.073742 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42"] Nov 03 17:56:34.079816 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:56:34.079783 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e3b992_04fc_4c47_b353_55a0fa75f7ed.slice/crio-f8252cd1eaf631222d3ef53e047e10fe81f081c1316111d130845b23904aef7d WatchSource:0}: Error finding container f8252cd1eaf631222d3ef53e047e10fe81f081c1316111d130845b23904aef7d: Status 404 returned error can't find the container with id f8252cd1eaf631222d3ef53e047e10fe81f081c1316111d130845b23904aef7d Nov 03 17:56:35.000012 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:34.999979 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" event={"ID":"c5e3b992-04fc-4c47-b353-55a0fa75f7ed","Type":"ContainerStarted","Data":"d0022d04c9948338905d71a5635142e39b84b7acc316113e3c6f932443be5882"} Nov 03 17:56:35.000157 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:35.000039 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" event={"ID":"c5e3b992-04fc-4c47-b353-55a0fa75f7ed","Type":"ContainerStarted","Data":"f8252cd1eaf631222d3ef53e047e10fe81f081c1316111d130845b23904aef7d"} Nov 03 17:56:35.000157 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:35.000126 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" Nov 03 17:56:35.168088 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:35.167967 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" podStartSLOduration=2.33464722 podStartE2EDuration="3.16795162s" totalImagesPullingTime="833.3044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:35.166972162 +0000 UTC m=+632.300601176" watchObservedRunningTime="2026-11-03 17:56:35.16795162 +0000 UTC m=+632.301580629" Nov 03 17:56:36.730266 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:36.730231 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:36.941215 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:36.941180 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:38.175248 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:38.175214 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:39.976282 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:39.976255 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg" Nov 03 17:56:39.976675 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:39.976437 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w" Nov 03 17:56:39.976675 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:39.976503 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9" Nov 03 17:56:41.011346 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:41.011318 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" Nov 03 17:56:41.871326 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:41.871294 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:42.090655 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:42.090584 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:42.476553 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:42.476523 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74"] Nov 03 17:56:42.480228 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:42.480207 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" Nov 03 17:56:42.574756 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:42.574728 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-0ccbb9-serving-cert\"" Nov 03 17:56:42.576602 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:42.576580 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74"] Nov 03 17:56:42.589659 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:42.589633 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac493e2-b140-4b42-93ca-fba0da0079de-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74\" (UID: \"0ac493e2-b140-4b42-93ca-fba0da0079de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" Nov 03 17:56:42.589774 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:42.589680 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac493e2-b140-4b42-93ca-fba0da0079de-maas-api-tls\") pod \"maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74\" (UID: \"0ac493e2-b140-4b42-93ca-fba0da0079de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" Nov 03 17:56:42.589868 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:42.589851 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac493e2-b140-4b42-93ca-fba0da0079de-metrics-certs\") pod \"maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74\" (UID: \"0ac493e2-b140-4b42-93ca-fba0da0079de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" Nov 03 17:56:42.590018 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:42.589998 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxzmn\" (UniqueName: \"kubernetes.io/projected/0ac493e2-b140-4b42-93ca-fba0da0079de-kube-api-access-jxzmn\") pod \"maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74\" (UID: \"0ac493e2-b140-4b42-93ca-fba0da0079de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" Nov 03 17:56:42.690772 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:42.690737 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac493e2-b140-4b42-93ca-fba0da0079de-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74\" (UID: \"0ac493e2-b140-4b42-93ca-fba0da0079de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" Nov 03 17:56:42.690964 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:42.690780 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac493e2-b140-4b42-93ca-fba0da0079de-maas-api-tls\") pod \"maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74\" (UID: \"0ac493e2-b140-4b42-93ca-fba0da0079de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" Nov 03 17:56:42.690964 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:42.690821 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac493e2-b140-4b42-93ca-fba0da0079de-metrics-certs\") pod \"maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74\" (UID: \"0ac493e2-b140-4b42-93ca-fba0da0079de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" Nov 03 17:56:42.690964 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:42.690911 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxzmn\" (UniqueName: \"kubernetes.io/projected/0ac493e2-b140-4b42-93ca-fba0da0079de-kube-api-access-jxzmn\") pod \"maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74\" (UID: \"0ac493e2-b140-4b42-93ca-fba0da0079de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" Nov 03 17:56:42.690964 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:56:42.690937 5805 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-serving-cert: secret "maas-api-e2e-worker-w0-0ccbb9-serving-cert" not found Nov 03 17:56:42.691182 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:56:42.691012 5805 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ac493e2-b140-4b42-93ca-fba0da0079de-maas-api-tls podName:0ac493e2-b140-4b42-93ca-fba0da0079de nodeName:}" failed. No retries permitted until 2026-11-03 17:56:43.190991152 +0000 UTC m=+640.324620154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ac493e2-b140-4b42-93ca-fba0da0079de-maas-api-tls") pod "maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" (UID: "0ac493e2-b140-4b42-93ca-fba0da0079de") : secret "maas-api-e2e-worker-w0-0ccbb9-serving-cert" not found Nov 03 17:56:42.691491 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:42.691468 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac493e2-b140-4b42-93ca-fba0da0079de-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74\" (UID: \"0ac493e2-b140-4b42-93ca-fba0da0079de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" Nov 03 17:56:42.693381 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:42.693358 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac493e2-b140-4b42-93ca-fba0da0079de-metrics-certs\") pod \"maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74\" (UID: \"0ac493e2-b140-4b42-93ca-fba0da0079de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" Nov 03 17:56:42.721920 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:42.721882 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:42.750757 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:42.750684 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxzmn\" (UniqueName: \"kubernetes.io/projected/0ac493e2-b140-4b42-93ca-fba0da0079de-kube-api-access-jxzmn\") pod \"maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74\" (UID: \"0ac493e2-b140-4b42-93ca-fba0da0079de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" Nov 03 17:56:42.928557 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:42.928526 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:43.127335 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:43.127302 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:43.195325 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:43.195293 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac493e2-b140-4b42-93ca-fba0da0079de-maas-api-tls\") pod \"maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74\" (UID: \"0ac493e2-b140-4b42-93ca-fba0da0079de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" Nov 03 17:56:43.197683 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:43.197656 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac493e2-b140-4b42-93ca-fba0da0079de-maas-api-tls\") pod \"maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74\" (UID: \"0ac493e2-b140-4b42-93ca-fba0da0079de\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" Nov 03 17:56:43.394829 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:43.394742 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" Nov 03 17:56:43.773893 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:43.773813 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74"] Nov 03 17:56:43.780133 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:56:43.780104 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac493e2_b140_4b42_93ca_fba0da0079de.slice/crio-796ef35819e0821e0257249ae3c1087e0335590c53b61bab92dfc171f1d4e487 WatchSource:0}: Error finding container 796ef35819e0821e0257249ae3c1087e0335590c53b61bab92dfc171f1d4e487: Status 404 returned error can't find the container with id 796ef35819e0821e0257249ae3c1087e0335590c53b61bab92dfc171f1d4e487 Nov 03 17:56:44.048873 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:44.048797 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" event={"ID":"0ac493e2-b140-4b42-93ca-fba0da0079de","Type":"ContainerStarted","Data":"796ef35819e0821e0257249ae3c1087e0335590c53b61bab92dfc171f1d4e487"} Nov 03 17:56:45.056172 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:45.056140 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" event={"ID":"0ac493e2-b140-4b42-93ca-fba0da0079de","Type":"ContainerStarted","Data":"ea3b995359065459a8d769075f688fdb27ef865e5fd0dbdac8380706af36308d"} Nov 03 17:56:45.056564 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:45.056263 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" Nov 03 17:56:45.168934 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:45.168876 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" podStartSLOduration=2.392634788 podStartE2EDuration="3.168860738s" totalImagesPullingTime="776.22595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:45.166685102 +0000 UTC m=+642.300314130" watchObservedRunningTime="2026-11-03 17:56:45.168860738 +0000 UTC m=+642.302489744" Nov 03 17:56:47.455526 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:47.455487 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:47.662607 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:47.662570 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:47.864094 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:47.864053 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:51.066345 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:51.066314 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" Nov 03 17:56:52.215385 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:52.215353 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:56:56.234294 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:56:56.234249 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:56:56.701203 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:56:56.697142 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:01.323298 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:01.323264 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:01.525031 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:01.524988 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:01.734264 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:01.734190 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:02.388272 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:02.388235 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:03.001751 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:03.001722 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:03.253014 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:03.252930 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:03.633529 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:03.633495 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:03.869664 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:03.869634 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:06.629595 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.629558 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:06.637747 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.637722 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l"] Nov 03 17:57:06.643079 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.643060 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 17:57:06.711604 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.711573 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-model-cache\") pod \"simulator-w1-ef9633-kserve-5867bfbd86-rjh4l\" (UID: \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\") " pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 17:57:06.711782 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.711696 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-home\") pod \"simulator-w1-ef9633-kserve-5867bfbd86-rjh4l\" (UID: \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\") " pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 17:57:06.711782 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.711736 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7gl9\" (UniqueName: \"kubernetes.io/projected/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-kube-api-access-s7gl9\") pod \"simulator-w1-ef9633-kserve-5867bfbd86-rjh4l\" (UID: \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\") " pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 17:57:06.711914 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.711862 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-tmp-dir\") pod \"simulator-w1-ef9633-kserve-5867bfbd86-rjh4l\" (UID: \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\") " pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 17:57:06.711914 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.711884 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-tls-certs\") pod \"simulator-w1-ef9633-kserve-5867bfbd86-rjh4l\" (UID: \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\") " pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 17:57:06.712062 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.711916 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-dshm\") pod \"simulator-w1-ef9633-kserve-5867bfbd86-rjh4l\" (UID: \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\") " pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 17:57:06.738179 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.738143 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-ef9633\"/\"default-dockercfg-6vdlf\"" Nov 03 17:57:06.738381 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.738363 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-ef9633\"/\"simulator-w1-ef9633-kserve-self-signed-certs\"" Nov 03 17:57:06.740161 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.740141 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l"] Nov 03 17:57:06.741338 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.741324 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-ef9633\"/\"openshift-service-ca.crt\"" Nov 03 17:57:06.741417 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.741354 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-ef9633\"/\"kube-root-ca.crt\"" Nov 03 17:57:06.813121 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.813089 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7gl9\" (UniqueName: \"kubernetes.io/projected/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-kube-api-access-s7gl9\") pod \"simulator-w1-ef9633-kserve-5867bfbd86-rjh4l\" (UID: \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\") " pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 17:57:06.813505 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.813164 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-tmp-dir\") pod \"simulator-w1-ef9633-kserve-5867bfbd86-rjh4l\" (UID: \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\") " pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 17:57:06.813505 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.813191 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-tls-certs\") pod \"simulator-w1-ef9633-kserve-5867bfbd86-rjh4l\" (UID: \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\") " pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 17:57:06.813505 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.813223 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-dshm\") pod \"simulator-w1-ef9633-kserve-5867bfbd86-rjh4l\" (UID: \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\") " pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 17:57:06.813505 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.813290 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-model-cache\") pod \"simulator-w1-ef9633-kserve-5867bfbd86-rjh4l\" (UID: \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\") " pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 17:57:06.813505 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.813387 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-home\") pod \"simulator-w1-ef9633-kserve-5867bfbd86-rjh4l\" (UID: \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\") " pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 17:57:06.813909 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.813878 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-tmp-dir\") pod \"simulator-w1-ef9633-kserve-5867bfbd86-rjh4l\" (UID: \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\") " pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 17:57:06.814195 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.813891 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-home\") pod \"simulator-w1-ef9633-kserve-5867bfbd86-rjh4l\" (UID: \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\") " pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 17:57:06.814195 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.814071 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-model-cache\") pod \"simulator-w1-ef9633-kserve-5867bfbd86-rjh4l\" (UID: \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\") " pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 17:57:06.816071 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.816046 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-dshm\") pod \"simulator-w1-ef9633-kserve-5867bfbd86-rjh4l\" (UID: \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\") " pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 17:57:06.816284 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.816263 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-tls-certs\") pod \"simulator-w1-ef9633-kserve-5867bfbd86-rjh4l\" (UID: \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\") " pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 17:57:06.839404 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.839373 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:06.870997 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.870978 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7gl9\" (UniqueName: \"kubernetes.io/projected/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-kube-api-access-s7gl9\") pod \"simulator-w1-ef9633-kserve-5867bfbd86-rjh4l\" (UID: \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\") " pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 17:57:06.953480 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:06.953404 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 17:57:07.325631 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:07.325595 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l"] Nov 03 17:57:07.332591 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:57:07.332552 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0aaaea9_8307_44e7_95d9_d83f0a2b6aa2.slice/crio-5f85a7ad6ff9857bb5c5dffa2998d9668ece862cafab7c75add2aeb41b1c2b04 WatchSource:0}: Error finding container 5f85a7ad6ff9857bb5c5dffa2998d9668ece862cafab7c75add2aeb41b1c2b04: Status 404 returned error can't find the container with id 5f85a7ad6ff9857bb5c5dffa2998d9668ece862cafab7c75add2aeb41b1c2b04 Nov 03 17:57:08.175722 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:08.175685 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" event={"ID":"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2","Type":"ContainerStarted","Data":"5f85a7ad6ff9857bb5c5dffa2998d9668ece862cafab7c75add2aeb41b1c2b04"} Nov 03 17:57:08.736340 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:08.736300 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:10.190421 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:10.190324 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" event={"ID":"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2","Type":"ContainerStarted","Data":"4310621185f485d9d9e07507103cdab04be1e5c652c8b899fd6bcc423164f4e6"} Nov 03 17:57:10.300029 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:10.299980 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" podStartSLOduration=1.813080731 podStartE2EDuration="4.299966872s" totalImagesPullingTime="2.486886141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:10.299129475 +0000 UTC m=+667.432758483" watchObservedRunningTime="2026-11-03 17:57:10.299966872 +0000 UTC m=+667.433595890" Nov 03 17:57:10.953646 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:10.953616 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 17:57:11.171339 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:11.171303 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:12.069899 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:12.069860 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:14.537436 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:14.537374 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:15.183323 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:15.183286 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:15.394824 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:15.394794 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:16.067775 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:16.067746 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:16.954523 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:16.954490 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 17:57:16.966962 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:16.966937 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 17:57:17.239240 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:17.239159 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 17:57:18.605681 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:18.605649 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:19.288609 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:19.288572 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:19.519420 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:19.519372 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:21.397818 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:21.397787 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:25.890714 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:25.890673 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:26.235439 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:57:26.235340 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:57:29.031106 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:29.031069 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:29.721337 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:29.721305 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:29.977660 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:29.977579 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:30.857682 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:30.857642 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:33.226717 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:33.226638 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:33.899291 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:33.899262 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:34.102610 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:34.102578 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:35.774536 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:35.774503 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:39.773273 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:39.773237 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl"] Nov 03 17:57:39.777250 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:39.777228 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 17:57:39.833235 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:39.833204 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-a86228\"/\"kube-root-ca.crt\"" Nov 03 17:57:39.875180 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:39.875149 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-a86228\"/\"openshift-service-ca.crt\"" Nov 03 17:57:39.875385 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:39.875281 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a86228\"/\"simulator-w4-a86228-kserve-self-signed-certs\"" Nov 03 17:57:39.875385 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:39.875368 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a86228\"/\"default-dockercfg-dgttj\"" Nov 03 17:57:39.877319 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:39.877295 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl"] Nov 03 17:57:39.915803 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:39.915771 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:39.925837 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:39.925813 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-home\") pod \"simulator-w4-a86228-kserve-6d6685d47d-5pcvl\" (UID: \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\") " pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 17:57:39.925964 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:39.925861 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-dshm\") pod \"simulator-w4-a86228-kserve-6d6685d47d-5pcvl\" (UID: \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\") " pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 17:57:39.926021 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:39.925981 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-model-cache\") pod \"simulator-w4-a86228-kserve-6d6685d47d-5pcvl\" (UID: \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\") " pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 17:57:39.926021 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:39.926016 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-tls-certs\") pod \"simulator-w4-a86228-kserve-6d6685d47d-5pcvl\" (UID: \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\") " pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 17:57:39.926135 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:39.926116 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-tmp-dir\") pod \"simulator-w4-a86228-kserve-6d6685d47d-5pcvl\" (UID: \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\") " pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 17:57:39.926180 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:39.926151 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmrc9\" (UniqueName: \"kubernetes.io/projected/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-kube-api-access-vmrc9\") pod \"simulator-w4-a86228-kserve-6d6685d47d-5pcvl\" (UID: \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\") " pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 17:57:40.026676 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:40.026595 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-model-cache\") pod \"simulator-w4-a86228-kserve-6d6685d47d-5pcvl\" (UID: \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\") " pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 17:57:40.026676 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:40.026651 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-tls-certs\") pod \"simulator-w4-a86228-kserve-6d6685d47d-5pcvl\" (UID: \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\") " pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 17:57:40.026884 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:40.026722 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-tmp-dir\") pod \"simulator-w4-a86228-kserve-6d6685d47d-5pcvl\" (UID: \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\") " pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 17:57:40.026884 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:40.026758 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmrc9\" (UniqueName: \"kubernetes.io/projected/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-kube-api-access-vmrc9\") pod \"simulator-w4-a86228-kserve-6d6685d47d-5pcvl\" (UID: \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\") " pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 17:57:40.026884 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:40.026783 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-home\") pod \"simulator-w4-a86228-kserve-6d6685d47d-5pcvl\" (UID: \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\") " pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 17:57:40.026884 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:40.026832 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-dshm\") pod \"simulator-w4-a86228-kserve-6d6685d47d-5pcvl\" (UID: \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\") " pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 17:57:40.027091 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:40.027068 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-model-cache\") pod \"simulator-w4-a86228-kserve-6d6685d47d-5pcvl\" (UID: \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\") " pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 17:57:40.027151 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:40.027106 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-tmp-dir\") pod \"simulator-w4-a86228-kserve-6d6685d47d-5pcvl\" (UID: \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\") " pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 17:57:40.027342 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:40.027319 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-home\") pod \"simulator-w4-a86228-kserve-6d6685d47d-5pcvl\" (UID: \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\") " pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 17:57:40.028979 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:40.028959 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-dshm\") pod \"simulator-w4-a86228-kserve-6d6685d47d-5pcvl\" (UID: \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\") " pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 17:57:40.029263 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:40.029233 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-tls-certs\") pod \"simulator-w4-a86228-kserve-6d6685d47d-5pcvl\" (UID: \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\") " pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 17:57:40.086367 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:40.086339 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmrc9\" (UniqueName: \"kubernetes.io/projected/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-kube-api-access-vmrc9\") pod \"simulator-w4-a86228-kserve-6d6685d47d-5pcvl\" (UID: \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\") " pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 17:57:40.089210 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:40.089177 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 17:57:40.324155 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:40.324121 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:40.465962 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:40.465924 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl"] Nov 03 17:57:40.472646 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:57:40.472615 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82c4f5f3_b386_4ade_bfc9_d5de175e5bcc.slice/crio-dd92b896d60ff47f9570d0c34154028cfc130443aa534cd183324360dd343706 WatchSource:0}: Error finding container dd92b896d60ff47f9570d0c34154028cfc130443aa534cd183324360dd343706: Status 404 returned error can't find the container with id dd92b896d60ff47f9570d0c34154028cfc130443aa534cd183324360dd343706 Nov 03 17:57:41.353464 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:41.353428 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" event={"ID":"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc","Type":"ContainerStarted","Data":"75bb4310cfc973dbeee45d21daa4f00884d45f80a820bb5f384c5389c8a2d1be"} Nov 03 17:57:41.353963 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:41.353476 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" event={"ID":"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc","Type":"ContainerStarted","Data":"dd92b896d60ff47f9570d0c34154028cfc130443aa534cd183324360dd343706"} Nov 03 17:57:41.464830 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:41.464786 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" podStartSLOduration=2.464773846 podStartE2EDuration="2.464773846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:41.463154438 +0000 UTC m=+698.596783468" watchObservedRunningTime="2026-11-03 17:57:41.464773846 +0000 UTC m=+698.598402890" Nov 03 17:57:42.089970 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:42.089939 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 17:57:44.893447 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:44.893409 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:45.106506 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:45.106473 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:49.797229 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:49.797189 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:50.090165 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:50.090139 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 17:57:50.102603 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:50.102577 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 17:57:50.413044 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:50.412969 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 17:57:50.540282 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:50.540247 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:51.229607 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:51.229538 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:51.444430 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:51.444382 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:52.332388 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:52.332359 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:54.689434 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:54.689376 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:57:56.235894 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:57:56.235853 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:57:59.370186 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:57:59.370148 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:58:03.966469 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:03.966426 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:58:06.929632 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:06.929598 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz"] Nov 03 17:58:06.933622 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:06.933585 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 17:58:06.974242 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:06.974217 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6glml\" (UniqueName: \"kubernetes.io/projected/ad8cc84a-716a-44c7-a257-1f630c48e1b6-kube-api-access-6glml\") pod \"simulator-w2-f79c02-kserve-5fccb59878-j9hvz\" (UID: \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\") " pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 17:58:06.974405 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:06.974270 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-tmp-dir\") pod \"simulator-w2-f79c02-kserve-5fccb59878-j9hvz\" (UID: \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\") " pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 17:58:06.974405 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:06.974300 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-home\") pod \"simulator-w2-f79c02-kserve-5fccb59878-j9hvz\" (UID: \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\") " pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 17:58:06.974540 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:06.974436 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8cc84a-716a-44c7-a257-1f630c48e1b6-tls-certs\") pod \"simulator-w2-f79c02-kserve-5fccb59878-j9hvz\" (UID: \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\") " pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 17:58:06.974540 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:06.974480 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-dshm\") pod \"simulator-w2-f79c02-kserve-5fccb59878-j9hvz\" (UID: \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\") " pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 17:58:06.974540 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:06.974499 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-model-cache\") pod \"simulator-w2-f79c02-kserve-5fccb59878-j9hvz\" (UID: \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\") " pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 17:58:06.999327 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:06.999296 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f79c02\"/\"kube-root-ca.crt\"" Nov 03 17:58:07.028554 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:07.028529 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f79c02\"/\"simulator-w2-f79c02-kserve-self-signed-certs\"" Nov 03 17:58:07.028710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:07.028559 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f79c02\"/\"openshift-service-ca.crt\"" Nov 03 17:58:07.028710 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:07.028562 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f79c02\"/\"default-dockercfg-pxq8r\"" Nov 03 17:58:07.030630 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:07.030606 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz"] Nov 03 17:58:07.075349 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:07.075310 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:58:07.077472 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:07.077442 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6glml\" (UniqueName: \"kubernetes.io/projected/ad8cc84a-716a-44c7-a257-1f630c48e1b6-kube-api-access-6glml\") pod \"simulator-w2-f79c02-kserve-5fccb59878-j9hvz\" (UID: \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\") " pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 17:58:07.077624 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:07.077517 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-tmp-dir\") pod \"simulator-w2-f79c02-kserve-5fccb59878-j9hvz\" (UID: \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\") " pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 17:58:07.077624 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:07.077555 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-home\") pod \"simulator-w2-f79c02-kserve-5fccb59878-j9hvz\" (UID: \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\") " pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 17:58:07.077624 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:07.077611 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8cc84a-716a-44c7-a257-1f630c48e1b6-tls-certs\") pod \"simulator-w2-f79c02-kserve-5fccb59878-j9hvz\" (UID: \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\") " pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 17:58:07.077799 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:07.077673 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-dshm\") pod \"simulator-w2-f79c02-kserve-5fccb59878-j9hvz\" (UID: \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\") " pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 17:58:07.077799 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:07.077695 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-model-cache\") pod \"simulator-w2-f79c02-kserve-5fccb59878-j9hvz\" (UID: \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\") " pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 17:58:07.078155 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:07.078129 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-model-cache\") pod \"simulator-w2-f79c02-kserve-5fccb59878-j9hvz\" (UID: \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\") " pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 17:58:07.078155 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:07.078148 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-home\") pod \"simulator-w2-f79c02-kserve-5fccb59878-j9hvz\" (UID: \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\") " pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 17:58:07.078619 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:07.078596 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-tmp-dir\") pod \"simulator-w2-f79c02-kserve-5fccb59878-j9hvz\" (UID: \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\") " pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 17:58:07.081504 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:07.081479 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-dshm\") pod \"simulator-w2-f79c02-kserve-5fccb59878-j9hvz\" (UID: \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\") " pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 17:58:07.081839 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:07.081809 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8cc84a-716a-44c7-a257-1f630c48e1b6-tls-certs\") pod \"simulator-w2-f79c02-kserve-5fccb59878-j9hvz\" (UID: \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\") " pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 17:58:07.135296 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:07.135260 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6glml\" (UniqueName: \"kubernetes.io/projected/ad8cc84a-716a-44c7-a257-1f630c48e1b6-kube-api-access-6glml\") pod \"simulator-w2-f79c02-kserve-5fccb59878-j9hvz\" (UID: \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\") " pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 17:58:07.245789 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:07.245705 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 17:58:07.625597 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:07.625559 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz"] Nov 03 17:58:07.631646 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:58:07.631619 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad8cc84a_716a_44c7_a257_1f630c48e1b6.slice/crio-6b6613a89a7c67c24b8b6e45af1eb43c1a1ed4619d5860e73159f885d1c5a5bb WatchSource:0}: Error finding container 6b6613a89a7c67c24b8b6e45af1eb43c1a1ed4619d5860e73159f885d1c5a5bb: Status 404 returned error can't find the container with id 6b6613a89a7c67c24b8b6e45af1eb43c1a1ed4619d5860e73159f885d1c5a5bb Nov 03 17:58:08.461832 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:08.461787 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:58:08.503004 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:08.502968 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" event={"ID":"ad8cc84a-716a-44c7-a257-1f630c48e1b6","Type":"ContainerStarted","Data":"93bb816e462ef068a6af9188705606d14ac355598cfa13d717696e17d82e3bfd"} Nov 03 17:58:08.503171 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:08.503011 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" event={"ID":"ad8cc84a-716a-44c7-a257-1f630c48e1b6","Type":"ContainerStarted","Data":"6b6613a89a7c67c24b8b6e45af1eb43c1a1ed4619d5860e73159f885d1c5a5bb"} Nov 03 17:58:08.615070 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:08.615014 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" podStartSLOduration=2.615000349 podStartE2EDuration="2.615000349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:08.613223385 +0000 UTC m=+725.746852415" watchObservedRunningTime="2026-11-03 17:58:08.615000349 +0000 UTC m=+725.748629393" Nov 03 17:58:09.089471 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:09.089438 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:58:09.246289 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:09.246254 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 17:58:12.999166 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:12.998665 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:58:16.995614 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:16.995576 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:58:17.246873 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:17.246785 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 17:58:17.259882 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:17.259854 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 17:58:17.511510 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:17.511417 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:58:17.565583 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:17.565556 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 17:58:20.671230 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:20.671199 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:58:22.057248 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:22.057215 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:58:26.235225 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:58:26.235187 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:58:26.520113 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:26.520038 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:58:31.053877 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:31.053842 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:58:34.593967 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.593934 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r"] Nov 03 17:58:34.597895 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.597868 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 17:58:34.646004 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.645960 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw6sp\" (UniqueName: \"kubernetes.io/projected/c968ad12-4333-4704-94f9-e4d34b204c81-kube-api-access-tw6sp\") pod \"simulator-w3-0adcbc-kserve-77d9b6c945-sh69r\" (UID: \"c968ad12-4333-4704-94f9-e4d34b204c81\") " pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 17:58:34.646004 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.646000 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-tmp-dir\") pod \"simulator-w3-0adcbc-kserve-77d9b6c945-sh69r\" (UID: \"c968ad12-4333-4704-94f9-e4d34b204c81\") " pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 17:58:34.646226 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.646100 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c968ad12-4333-4704-94f9-e4d34b204c81-tls-certs\") pod \"simulator-w3-0adcbc-kserve-77d9b6c945-sh69r\" (UID: \"c968ad12-4333-4704-94f9-e4d34b204c81\") " pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 17:58:34.646226 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.646154 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-home\") pod \"simulator-w3-0adcbc-kserve-77d9b6c945-sh69r\" (UID: \"c968ad12-4333-4704-94f9-e4d34b204c81\") " pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 17:58:34.646315 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.646300 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-dshm\") pod \"simulator-w3-0adcbc-kserve-77d9b6c945-sh69r\" (UID: \"c968ad12-4333-4704-94f9-e4d34b204c81\") " pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 17:58:34.646362 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.646327 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-model-cache\") pod \"simulator-w3-0adcbc-kserve-77d9b6c945-sh69r\" (UID: \"c968ad12-4333-4704-94f9-e4d34b204c81\") " pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 17:58:34.647047 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.647025 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-0adcbc\"/\"openshift-service-ca.crt\"" Nov 03 17:58:34.692907 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.692882 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-0adcbc\"/\"kube-root-ca.crt\"" Nov 03 17:58:34.693426 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.693406 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-0adcbc\"/\"default-dockercfg-tcxw9\"" Nov 03 17:58:34.693559 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.693464 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-0adcbc\"/\"simulator-w3-0adcbc-kserve-self-signed-certs\"" Nov 03 17:58:34.695611 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.695588 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r"] Nov 03 17:58:34.747670 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.747643 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-home\") pod \"simulator-w3-0adcbc-kserve-77d9b6c945-sh69r\" (UID: \"c968ad12-4333-4704-94f9-e4d34b204c81\") " pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 17:58:34.747831 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.747754 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-dshm\") pod \"simulator-w3-0adcbc-kserve-77d9b6c945-sh69r\" (UID: \"c968ad12-4333-4704-94f9-e4d34b204c81\") " pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 17:58:34.747831 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.747777 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-model-cache\") pod \"simulator-w3-0adcbc-kserve-77d9b6c945-sh69r\" (UID: \"c968ad12-4333-4704-94f9-e4d34b204c81\") " pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 17:58:34.747831 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.747813 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw6sp\" (UniqueName: \"kubernetes.io/projected/c968ad12-4333-4704-94f9-e4d34b204c81-kube-api-access-tw6sp\") pod \"simulator-w3-0adcbc-kserve-77d9b6c945-sh69r\" (UID: \"c968ad12-4333-4704-94f9-e4d34b204c81\") " pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 17:58:34.748000 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.747837 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-tmp-dir\") pod \"simulator-w3-0adcbc-kserve-77d9b6c945-sh69r\" (UID: \"c968ad12-4333-4704-94f9-e4d34b204c81\") " pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 17:58:34.748000 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.747881 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c968ad12-4333-4704-94f9-e4d34b204c81-tls-certs\") pod \"simulator-w3-0adcbc-kserve-77d9b6c945-sh69r\" (UID: \"c968ad12-4333-4704-94f9-e4d34b204c81\") " pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 17:58:34.748184 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.748080 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-home\") pod \"simulator-w3-0adcbc-kserve-77d9b6c945-sh69r\" (UID: \"c968ad12-4333-4704-94f9-e4d34b204c81\") " pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 17:58:34.748255 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.748231 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-model-cache\") pod \"simulator-w3-0adcbc-kserve-77d9b6c945-sh69r\" (UID: \"c968ad12-4333-4704-94f9-e4d34b204c81\") " pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 17:58:34.748522 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.748500 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-tmp-dir\") pod \"simulator-w3-0adcbc-kserve-77d9b6c945-sh69r\" (UID: \"c968ad12-4333-4704-94f9-e4d34b204c81\") " pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 17:58:34.750093 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.750069 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-dshm\") pod \"simulator-w3-0adcbc-kserve-77d9b6c945-sh69r\" (UID: \"c968ad12-4333-4704-94f9-e4d34b204c81\") " pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 17:58:34.750238 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.750220 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c968ad12-4333-4704-94f9-e4d34b204c81-tls-certs\") pod \"simulator-w3-0adcbc-kserve-77d9b6c945-sh69r\" (UID: \"c968ad12-4333-4704-94f9-e4d34b204c81\") " pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 17:58:34.814433 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.814378 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw6sp\" (UniqueName: \"kubernetes.io/projected/c968ad12-4333-4704-94f9-e4d34b204c81-kube-api-access-tw6sp\") pod \"simulator-w3-0adcbc-kserve-77d9b6c945-sh69r\" (UID: \"c968ad12-4333-4704-94f9-e4d34b204c81\") " pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 17:58:34.860470 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.860385 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:58:34.910582 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:34.910548 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 17:58:35.291985 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:35.291905 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r"] Nov 03 17:58:35.298724 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:58:35.298695 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc968ad12_4333_4704_94f9_e4d34b204c81.slice/crio-422e70a148ccff2876b9943e197cf2ccfc1cb943416363a256e431494467b00a WatchSource:0}: Error finding container 422e70a148ccff2876b9943e197cf2ccfc1cb943416363a256e431494467b00a: Status 404 returned error can't find the container with id 422e70a148ccff2876b9943e197cf2ccfc1cb943416363a256e431494467b00a Nov 03 17:58:35.650770 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:35.650738 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" event={"ID":"c968ad12-4333-4704-94f9-e4d34b204c81","Type":"ContainerStarted","Data":"ed963466efb705d90a5ba7358261a0b54e4a22282ea43b01eccc2c36a9a648e4"} Nov 03 17:58:35.651139 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:35.650781 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" event={"ID":"c968ad12-4333-4704-94f9-e4d34b204c81","Type":"ContainerStarted","Data":"422e70a148ccff2876b9943e197cf2ccfc1cb943416363a256e431494467b00a"} Nov 03 17:58:35.678357 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:35.678329 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:58:35.770924 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:35.770873 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" podStartSLOduration=1.7708596380000001 podStartE2EDuration="1.770859638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:35.767973489 +0000 UTC m=+752.901602497" watchObservedRunningTime="2026-11-03 17:58:35.770859638 +0000 UTC m=+752.904488659" Nov 03 17:58:35.911083 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:35.911005 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 17:58:37.774023 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:37.773993 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:58:40.267112 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:40.267082 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:58:44.760604 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:44.760557 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:58:44.911609 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:44.911581 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 17:58:44.930050 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:44.930025 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 17:58:45.711559 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:45.711530 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 17:58:49.418169 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:49.418132 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:58:53.972247 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:53.972214 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:58:54.573754 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:54.573725 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:58:56.236993 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:58:56.236959 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:58:58.469273 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:58:58.469237 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:59:03.084387 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:03.084349 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:59:07.580467 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:07.580430 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:59:09.428965 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:09.428909 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:59:12.153360 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:12.153330 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:59:16.660620 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:16.660588 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:59:20.118991 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:20.118887 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:59:21.166779 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:21.166742 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:59:25.674329 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:25.674285 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:59:26.235149 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:59:26.235110 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:59:26.651052 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.651019 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8"] Nov 03 17:59:26.655000 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.654976 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 17:59:26.731522 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.731494 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-dshm\") pod \"premium-w0-0ccbb9-kserve-dcfc69489-kczk8\" (UID: \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\") " pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 17:59:26.731951 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.731528 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-home\") pod \"premium-w0-0ccbb9-kserve-dcfc69489-kczk8\" (UID: \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\") " pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 17:59:26.731951 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.731624 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-tmp-dir\") pod \"premium-w0-0ccbb9-kserve-dcfc69489-kczk8\" (UID: \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\") " pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 17:59:26.731951 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.731674 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-model-cache\") pod \"premium-w0-0ccbb9-kserve-dcfc69489-kczk8\" (UID: \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\") " pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 17:59:26.731951 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.731839 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj2pv\" (UniqueName: \"kubernetes.io/projected/43eecb04-1091-4dde-9b9c-42a0fb28f08f-kube-api-access-jj2pv\") pod \"premium-w0-0ccbb9-kserve-dcfc69489-kczk8\" (UID: \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\") " pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 17:59:26.731951 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.731892 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43eecb04-1091-4dde-9b9c-42a0fb28f08f-tls-certs\") pod \"premium-w0-0ccbb9-kserve-dcfc69489-kczk8\" (UID: \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\") " pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 17:59:26.749904 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.749878 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0ccbb9\"/\"kube-root-ca.crt\"" Nov 03 17:59:26.749904 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.749900 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-0ccbb9\"/\"openshift-service-ca.crt\"" Nov 03 17:59:26.750090 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.749975 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0ccbb9\"/\"default-dockercfg-bw54c\"" Nov 03 17:59:26.750090 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.749976 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0ccbb9\"/\"premium-w0-0ccbb9-kserve-self-signed-certs\"" Nov 03 17:59:26.752169 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.752150 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8"] Nov 03 17:59:26.801771 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.801743 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:59:26.832625 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.832594 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-model-cache\") pod \"premium-w0-0ccbb9-kserve-dcfc69489-kczk8\" (UID: \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\") " pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 17:59:26.832777 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.832681 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj2pv\" (UniqueName: \"kubernetes.io/projected/43eecb04-1091-4dde-9b9c-42a0fb28f08f-kube-api-access-jj2pv\") pod \"premium-w0-0ccbb9-kserve-dcfc69489-kczk8\" (UID: \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\") " pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 17:59:26.832777 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.832713 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43eecb04-1091-4dde-9b9c-42a0fb28f08f-tls-certs\") pod \"premium-w0-0ccbb9-kserve-dcfc69489-kczk8\" (UID: \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\") " pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 17:59:26.832777 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.832736 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-dshm\") pod \"premium-w0-0ccbb9-kserve-dcfc69489-kczk8\" (UID: \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\") " pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 17:59:26.832777 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.832751 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-home\") pod \"premium-w0-0ccbb9-kserve-dcfc69489-kczk8\" (UID: \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\") " pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 17:59:26.832982 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.832953 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-tmp-dir\") pod \"premium-w0-0ccbb9-kserve-dcfc69489-kczk8\" (UID: \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\") " pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 17:59:26.833103 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.833082 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-model-cache\") pod \"premium-w0-0ccbb9-kserve-dcfc69489-kczk8\" (UID: \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\") " pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 17:59:26.833178 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.833099 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-home\") pod \"premium-w0-0ccbb9-kserve-dcfc69489-kczk8\" (UID: \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\") " pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 17:59:26.833263 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.833244 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-tmp-dir\") pod \"premium-w0-0ccbb9-kserve-dcfc69489-kczk8\" (UID: \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\") " pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 17:59:26.835010 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.834990 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-dshm\") pod \"premium-w0-0ccbb9-kserve-dcfc69489-kczk8\" (UID: \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\") " pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 17:59:26.835274 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.835256 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43eecb04-1091-4dde-9b9c-42a0fb28f08f-tls-certs\") pod \"premium-w0-0ccbb9-kserve-dcfc69489-kczk8\" (UID: \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\") " pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 17:59:26.890043 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.890016 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj2pv\" (UniqueName: \"kubernetes.io/projected/43eecb04-1091-4dde-9b9c-42a0fb28f08f-kube-api-access-jj2pv\") pod \"premium-w0-0ccbb9-kserve-dcfc69489-kczk8\" (UID: \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\") " pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 17:59:26.966228 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:26.966156 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 17:59:27.354908 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:27.354876 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8"] Nov 03 17:59:27.361541 ip-10-0-137-164 kubenswrapper[5805]: W1103 17:59:27.361515 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43eecb04_1091_4dde_9b9c_42a0fb28f08f.slice/crio-ed0b4fd4b2fb194ea4282eda5de9a76bded8f8cbb57d48fd961d4096218bbd7e WatchSource:0}: Error finding container ed0b4fd4b2fb194ea4282eda5de9a76bded8f8cbb57d48fd961d4096218bbd7e: Status 404 returned error can't find the container with id ed0b4fd4b2fb194ea4282eda5de9a76bded8f8cbb57d48fd961d4096218bbd7e Nov 03 17:59:27.916572 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:27.916532 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" event={"ID":"43eecb04-1091-4dde-9b9c-42a0fb28f08f","Type":"ContainerStarted","Data":"5e603a64e0fa63996c250c63a20f8e96e690b8ffa7220370c0cea00a274492fe"} Nov 03 17:59:27.916572 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:27.916572 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" event={"ID":"43eecb04-1091-4dde-9b9c-42a0fb28f08f","Type":"ContainerStarted","Data":"ed0b4fd4b2fb194ea4282eda5de9a76bded8f8cbb57d48fd961d4096218bbd7e"} Nov 03 17:59:27.966644 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:27.966618 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 17:59:28.027496 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:28.027444 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" podStartSLOduration=2.027429352 podStartE2EDuration="2.027429352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:59:28.026579555 +0000 UTC m=+805.160208581" watchObservedRunningTime="2026-11-03 17:59:28.027429352 +0000 UTC m=+805.161058362" Nov 03 17:59:30.267099 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:30.267059 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:59:34.217909 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:34.217877 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:59:34.908408 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:34.908370 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:59:36.968741 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:36.968703 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 17:59:36.981373 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:36.981348 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 17:59:37.977912 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:37.977881 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 17:59:39.496456 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:39.496382 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:59:44.309245 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:44.309163 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:59:44.968469 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:44.968420 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:59:49.052243 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:49.052214 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:59:53.679841 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:53.679795 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 17:59:56.235460 ip-10-0-137-164 kubenswrapper[5805]: E1103 17:59:56.235347 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 17:59:58.329428 ip-10-0-137-164 kubenswrapper[5805]: I1103 17:59:58.329342 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:00:00.176680 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.176647 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2"] Nov 03 18:00:00.180651 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.180628 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt"] Nov 03 18:00:00.180797 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.180771 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2" Nov 03 18:00:00.184547 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.184521 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt" Nov 03 18:00:00.199169 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.199142 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv"] Nov 03 18:00:00.202646 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.202626 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv" Nov 03 18:00:00.229960 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.229936 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\"" Nov 03 18:00:00.235938 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.235897 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2"] Nov 03 18:00:00.268558 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.268530 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7ntn\" (UniqueName: \"kubernetes.io/projected/7ea465a0-b086-41d0-ace1-e47518c0c6c7-kube-api-access-t7ntn\") pod \"maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt\" (UID: \"7ea465a0-b086-41d0-ace1-e47518c0c6c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt" Nov 03 18:00:00.268739 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.268579 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssm5b\" (UniqueName: \"kubernetes.io/projected/c2477be6-8df0-4e5b-bbd1-3c7fffcc747d-kube-api-access-ssm5b\") pod \"maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv\" (UID: \"c2477be6-8df0-4e5b-bbd1-3c7fffcc747d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv" Nov 03 18:00:00.268739 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.268600 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e32b3864-f093-472d-a783-ea1c4f570b80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2\" (UID: \"e32b3864-f093-472d-a783-ea1c4f570b80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2" Nov 03 18:00:00.268870 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.268843 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9nvc\" (UniqueName: \"kubernetes.io/projected/e32b3864-f093-472d-a783-ea1c4f570b80-kube-api-access-p9nvc\") pod \"maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2\" (UID: \"e32b3864-f093-472d-a783-ea1c4f570b80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2" Nov 03 18:00:00.268986 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.268963 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2477be6-8df0-4e5b-bbd1-3c7fffcc747d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv\" (UID: \"c2477be6-8df0-4e5b-bbd1-3c7fffcc747d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv" Nov 03 18:00:00.269082 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.269057 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea465a0-b086-41d0-ace1-e47518c0c6c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt\" (UID: \"7ea465a0-b086-41d0-ace1-e47518c0c6c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt" Nov 03 18:00:00.271785 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.271757 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt"] Nov 03 18:00:00.336631 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.336596 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv"] Nov 03 18:00:00.341513 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.341483 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:00:00.369853 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.369822 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9nvc\" (UniqueName: \"kubernetes.io/projected/e32b3864-f093-472d-a783-ea1c4f570b80-kube-api-access-p9nvc\") pod \"maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2\" (UID: \"e32b3864-f093-472d-a783-ea1c4f570b80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2" Nov 03 18:00:00.369998 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.369885 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2477be6-8df0-4e5b-bbd1-3c7fffcc747d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv\" (UID: \"c2477be6-8df0-4e5b-bbd1-3c7fffcc747d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv" Nov 03 18:00:00.369998 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.369959 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea465a0-b086-41d0-ace1-e47518c0c6c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt\" (UID: \"7ea465a0-b086-41d0-ace1-e47518c0c6c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt" Nov 03 18:00:00.369998 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.369989 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7ntn\" (UniqueName: \"kubernetes.io/projected/7ea465a0-b086-41d0-ace1-e47518c0c6c7-kube-api-access-t7ntn\") pod \"maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt\" (UID: \"7ea465a0-b086-41d0-ace1-e47518c0c6c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt" Nov 03 18:00:00.370149 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.370032 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssm5b\" (UniqueName: \"kubernetes.io/projected/c2477be6-8df0-4e5b-bbd1-3c7fffcc747d-kube-api-access-ssm5b\") pod \"maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv\" (UID: \"c2477be6-8df0-4e5b-bbd1-3c7fffcc747d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv" Nov 03 18:00:00.370149 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.370060 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e32b3864-f093-472d-a783-ea1c4f570b80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2\" (UID: \"e32b3864-f093-472d-a783-ea1c4f570b80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2" Nov 03 18:00:00.370698 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.370674 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea465a0-b086-41d0-ace1-e47518c0c6c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt\" (UID: \"7ea465a0-b086-41d0-ace1-e47518c0c6c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt" Nov 03 18:00:00.370799 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.370705 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e32b3864-f093-472d-a783-ea1c4f570b80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2\" (UID: \"e32b3864-f093-472d-a783-ea1c4f570b80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2" Nov 03 18:00:00.370799 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.370780 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2477be6-8df0-4e5b-bbd1-3c7fffcc747d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv\" (UID: \"c2477be6-8df0-4e5b-bbd1-3c7fffcc747d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv" Nov 03 18:00:00.428219 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.428151 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssm5b\" (UniqueName: \"kubernetes.io/projected/c2477be6-8df0-4e5b-bbd1-3c7fffcc747d-kube-api-access-ssm5b\") pod \"maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv\" (UID: \"c2477be6-8df0-4e5b-bbd1-3c7fffcc747d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv" Nov 03 18:00:00.428366 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.428258 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9nvc\" (UniqueName: \"kubernetes.io/projected/e32b3864-f093-472d-a783-ea1c4f570b80-kube-api-access-p9nvc\") pod \"maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2\" (UID: \"e32b3864-f093-472d-a783-ea1c4f570b80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2" Nov 03 18:00:00.428366 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.428308 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7ntn\" (UniqueName: \"kubernetes.io/projected/7ea465a0-b086-41d0-ace1-e47518c0c6c7-kube-api-access-t7ntn\") pod \"maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt\" (UID: \"7ea465a0-b086-41d0-ace1-e47518c0c6c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt" Nov 03 18:00:00.496204 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.496172 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2" Nov 03 18:00:00.502516 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.502495 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt" Nov 03 18:00:00.521509 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.521481 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv" Nov 03 18:00:00.890274 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.890230 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2"] Nov 03 18:00:00.896343 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:00:00.896314 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32b3864_f093_472d_a783_ea1c4f570b80.slice/crio-9fc65c62b89c8eca0c17ac235b1d87054ad388cfb0ef1708ee06230a754d10d7 WatchSource:0}: Error finding container 9fc65c62b89c8eca0c17ac235b1d87054ad388cfb0ef1708ee06230a754d10d7: Status 404 returned error can't find the container with id 9fc65c62b89c8eca0c17ac235b1d87054ad388cfb0ef1708ee06230a754d10d7 Nov 03 18:00:00.942521 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.942491 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt"] Nov 03 18:00:00.946069 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:00.946054 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv"] Nov 03 18:00:00.949201 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:00:00.949174 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea465a0_b086_41d0_ace1_e47518c0c6c7.slice/crio-7f4833d8111524f34ba4225cc4330bd93ff7c2173e0578eb3e92da2a2ba037fc WatchSource:0}: Error finding container 7f4833d8111524f34ba4225cc4330bd93ff7c2173e0578eb3e92da2a2ba037fc: Status 404 returned error can't find the container with id 7f4833d8111524f34ba4225cc4330bd93ff7c2173e0578eb3e92da2a2ba037fc Nov 03 18:00:00.952284 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:00:00.952233 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2477be6_8df0_4e5b_bbd1_3c7fffcc747d.slice/crio-abfabc199870be44bcd54c6d86bb31926527c077f74e3422fd94334cce81507b WatchSource:0}: Error finding container abfabc199870be44bcd54c6d86bb31926527c077f74e3422fd94334cce81507b: Status 404 returned error can't find the container with id abfabc199870be44bcd54c6d86bb31926527c077f74e3422fd94334cce81507b Nov 03 18:00:01.109187 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:01.109152 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv" event={"ID":"c2477be6-8df0-4e5b-bbd1-3c7fffcc747d","Type":"ContainerStarted","Data":"abfabc199870be44bcd54c6d86bb31926527c077f74e3422fd94334cce81507b"} Nov 03 18:00:01.110319 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:01.110293 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt" event={"ID":"7ea465a0-b086-41d0-ace1-e47518c0c6c7","Type":"ContainerStarted","Data":"7f4833d8111524f34ba4225cc4330bd93ff7c2173e0578eb3e92da2a2ba037fc"} Nov 03 18:00:01.111444 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:01.111422 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2" event={"ID":"e32b3864-f093-472d-a783-ea1c4f570b80","Type":"ContainerStarted","Data":"9fc65c62b89c8eca0c17ac235b1d87054ad388cfb0ef1708ee06230a754d10d7"} Nov 03 18:00:02.893165 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:02.893130 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:00:06.146368 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:06.146336 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv" event={"ID":"c2477be6-8df0-4e5b-bbd1-3c7fffcc747d","Type":"ContainerStarted","Data":"42be887314bb8ec43f94279d72387ab034b75246fccabfcb49d9287c35c3cb6f"} Nov 03 18:00:06.273643 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:06.273589 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv" podStartSLOduration=1.878704017 podStartE2EDuration="6.273576004s" totalImagesPullingTime="4.394871987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:06.270919886 +0000 UTC m=+843.404548893" watchObservedRunningTime="2026-11-03 18:00:06.273576004 +0000 UTC m=+843.407205014" Nov 03 18:00:06.285426 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:06.285384 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644"] Nov 03 18:00:06.285617 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:06.285596 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" podUID="293f67c8-c8cc-4198-8e03-79c4ae3a710c" containerName="istio-proxy" containerID="cri-o://9c63ffcc383c8a7996bde478d980c6bb632293d4596dfd5a6272d164b901d810" gracePeriod=30 Nov 03 18:00:07.161596 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:07.161545 5805 generic.go:356] "Generic (PLEG): container finished" podID="c2477be6-8df0-4e5b-bbd1-3c7fffcc747d" containerID="42be887314bb8ec43f94279d72387ab034b75246fccabfcb49d9287c35c3cb6f" exitCode=60 Nov 03 18:00:07.162041 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:07.161633 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv" event={"ID":"c2477be6-8df0-4e5b-bbd1-3c7fffcc747d","Type":"ContainerDied","Data":"42be887314bb8ec43f94279d72387ab034b75246fccabfcb49d9287c35c3cb6f"} Nov 03 18:00:07.162041 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:07.161870 5805 scope.go:122] "RemoveContainer" containerID="42be887314bb8ec43f94279d72387ab034b75246fccabfcb49d9287c35c3cb6f" Nov 03 18:00:07.163648 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:07.163623 5805 generic.go:356] "Generic (PLEG): container finished" podID="7ea465a0-b086-41d0-ace1-e47518c0c6c7" containerID="68aa80b2b53bf405bb410249a847dc46abbec55f6e13b98ce728abf5c273465d" exitCode=60 Nov 03 18:00:07.163776 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:07.163667 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt" event={"ID":"7ea465a0-b086-41d0-ace1-e47518c0c6c7","Type":"ContainerDied","Data":"68aa80b2b53bf405bb410249a847dc46abbec55f6e13b98ce728abf5c273465d"} Nov 03 18:00:07.163919 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:07.163903 5805 scope.go:122] "RemoveContainer" containerID="68aa80b2b53bf405bb410249a847dc46abbec55f6e13b98ce728abf5c273465d" Nov 03 18:00:07.585996 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:07.585923 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:00:08.171052 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:08.171013 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2" event={"ID":"e32b3864-f093-472d-a783-ea1c4f570b80","Type":"ContainerStarted","Data":"ca5860a30c95f192688893afa54e223a5d4f160099acb3d06d59f1ba9e644581"} Nov 03 18:00:08.172800 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:08.172772 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv" event={"ID":"c2477be6-8df0-4e5b-bbd1-3c7fffcc747d","Type":"ContainerStarted","Data":"3428cca7c21a9a2ee1ad5fa374089df5fe57aed6ef18e276f5aa4f5c063b8b1a"} Nov 03 18:00:08.174463 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:08.174439 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt" event={"ID":"7ea465a0-b086-41d0-ace1-e47518c0c6c7","Type":"ContainerStarted","Data":"050a4916c0b7e15abc6f410df13a73f3fba4e8f5dc0a52d81c2187836470e5e7"} Nov 03 18:00:08.280138 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:08.280082 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2" podStartSLOduration=1.862493728 podStartE2EDuration="8.280063711s" totalImagesPullingTime="6.417569983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:08.278770298 +0000 UTC m=+845.412399326" watchObservedRunningTime="2026-11-03 18:00:08.280063711 +0000 UTC m=+845.413692722" Nov 03 18:00:08.501231 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:08.501129 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt" podStartSLOduration=3.487258647 podStartE2EDuration="8.501116295s" totalImagesPullingTime="5.013857648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:08.498676672 +0000 UTC m=+845.632305682" watchObservedRunningTime="2026-11-03 18:00:08.501116295 +0000 UTC m=+845.634745306" Nov 03 18:00:09.181545 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:09.181505 5805 generic.go:356] "Generic (PLEG): container finished" podID="c2477be6-8df0-4e5b-bbd1-3c7fffcc747d" containerID="3428cca7c21a9a2ee1ad5fa374089df5fe57aed6ef18e276f5aa4f5c063b8b1a" exitCode=60 Nov 03 18:00:09.182009 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:09.181606 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv" event={"ID":"c2477be6-8df0-4e5b-bbd1-3c7fffcc747d","Type":"ContainerDied","Data":"3428cca7c21a9a2ee1ad5fa374089df5fe57aed6ef18e276f5aa4f5c063b8b1a"} Nov 03 18:00:09.182009 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:09.181688 5805 scope.go:122] "RemoveContainer" containerID="42be887314bb8ec43f94279d72387ab034b75246fccabfcb49d9287c35c3cb6f" Nov 03 18:00:09.182009 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:09.181869 5805 scope.go:122] "RemoveContainer" containerID="3428cca7c21a9a2ee1ad5fa374089df5fe57aed6ef18e276f5aa4f5c063b8b1a" Nov 03 18:00:09.183825 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:09.183805 5805 generic.go:356] "Generic (PLEG): container finished" podID="7ea465a0-b086-41d0-ace1-e47518c0c6c7" containerID="050a4916c0b7e15abc6f410df13a73f3fba4e8f5dc0a52d81c2187836470e5e7" exitCode=60 Nov 03 18:00:09.183922 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:09.183873 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt" event={"ID":"7ea465a0-b086-41d0-ace1-e47518c0c6c7","Type":"ContainerDied","Data":"050a4916c0b7e15abc6f410df13a73f3fba4e8f5dc0a52d81c2187836470e5e7"} Nov 03 18:00:09.184111 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:09.184095 5805 scope.go:122] "RemoveContainer" containerID="050a4916c0b7e15abc6f410df13a73f3fba4e8f5dc0a52d81c2187836470e5e7" Nov 03 18:00:09.184408 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:00:09.184371 5805 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt_odh-ai-gateway-infra(7ea465a0-b086-41d0-ace1-e47518c0c6c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt" podUID="7ea465a0-b086-41d0-ace1-e47518c0c6c7" Nov 03 18:00:09.184784 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:00:09.184733 5805 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv_odh-ai-gateway-infra(c2477be6-8df0-4e5b-bbd1-3c7fffcc747d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv" podUID="c2477be6-8df0-4e5b-bbd1-3c7fffcc747d" Nov 03 18:00:09.185774 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:09.185704 5805 generic.go:356] "Generic (PLEG): container finished" podID="e32b3864-f093-472d-a783-ea1c4f570b80" containerID="ca5860a30c95f192688893afa54e223a5d4f160099acb3d06d59f1ba9e644581" exitCode=60 Nov 03 18:00:09.185864 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:09.185734 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2" event={"ID":"e32b3864-f093-472d-a783-ea1c4f570b80","Type":"ContainerDied","Data":"ca5860a30c95f192688893afa54e223a5d4f160099acb3d06d59f1ba9e644581"} Nov 03 18:00:09.185961 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:09.185946 5805 scope.go:122] "RemoveContainer" containerID="ca5860a30c95f192688893afa54e223a5d4f160099acb3d06d59f1ba9e644581" Nov 03 18:00:09.194715 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:09.194695 5805 scope.go:122] "RemoveContainer" containerID="68aa80b2b53bf405bb410249a847dc46abbec55f6e13b98ce728abf5c273465d" Nov 03 18:00:09.765793 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:09.765701 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:00:10.195339 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:10.195308 5805 generic.go:356] "Generic (PLEG): container finished" podID="e32b3864-f093-472d-a783-ea1c4f570b80" containerID="29e62254df6cb50ffc51cf980a433ab44fa1e69b0e32b7a4495a8ab6741f1b5d" exitCode=60 Nov 03 18:00:10.195824 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:10.195414 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2" event={"ID":"e32b3864-f093-472d-a783-ea1c4f570b80","Type":"ContainerDied","Data":"29e62254df6cb50ffc51cf980a433ab44fa1e69b0e32b7a4495a8ab6741f1b5d"} Nov 03 18:00:10.195824 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:10.195479 5805 scope.go:122] "RemoveContainer" containerID="ca5860a30c95f192688893afa54e223a5d4f160099acb3d06d59f1ba9e644581" Nov 03 18:00:10.195824 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:10.195795 5805 scope.go:122] "RemoveContainer" containerID="29e62254df6cb50ffc51cf980a433ab44fa1e69b0e32b7a4495a8ab6741f1b5d" Nov 03 18:00:10.196148 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:00:10.196113 5805 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2_odh-ai-gateway-infra(e32b3864-f093-472d-a783-ea1c4f570b80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2" podUID="e32b3864-f093-472d-a783-ea1c4f570b80" Nov 03 18:00:10.197562 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:10.197546 5805 scope.go:122] "RemoveContainer" containerID="3428cca7c21a9a2ee1ad5fa374089df5fe57aed6ef18e276f5aa4f5c063b8b1a" Nov 03 18:00:10.197806 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:00:10.197783 5805 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv_odh-ai-gateway-infra(c2477be6-8df0-4e5b-bbd1-3c7fffcc747d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv" podUID="c2477be6-8df0-4e5b-bbd1-3c7fffcc747d" Nov 03 18:00:10.199335 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:10.199320 5805 scope.go:122] "RemoveContainer" containerID="050a4916c0b7e15abc6f410df13a73f3fba4e8f5dc0a52d81c2187836470e5e7" Nov 03 18:00:10.199553 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:00:10.199534 5805 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt_odh-ai-gateway-infra(7ea465a0-b086-41d0-ace1-e47518c0c6c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt" podUID="7ea465a0-b086-41d0-ace1-e47518c0c6c7" Nov 03 18:00:11.206658 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.206629 5805 scope.go:122] "RemoveContainer" containerID="29e62254df6cb50ffc51cf980a433ab44fa1e69b0e32b7a4495a8ab6741f1b5d" Nov 03 18:00:11.207023 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:00:11.206925 5805 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2_odh-ai-gateway-infra(e32b3864-f093-472d-a783-ea1c4f570b80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2" podUID="e32b3864-f093-472d-a783-ea1c4f570b80" Nov 03 18:00:11.534191 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.534163 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 18:00:11.594578 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.594545 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-data\") pod \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " Nov 03 18:00:11.594774 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.594623 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-token\" (UniqueName: \"kubernetes.io/projected/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-token\") pod \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " Nov 03 18:00:11.594774 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.594651 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istiod-ca-cert\") pod \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " Nov 03 18:00:11.594774 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.594675 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/293f67c8-c8cc-4198-8e03-79c4ae3a710c-kube-api-access-b6jpf\" (UniqueName: \"kubernetes.io/projected/293f67c8-c8cc-4198-8e03-79c4ae3a710c-kube-api-access-b6jpf\") pod \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " Nov 03 18:00:11.594774 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.594758 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-credential-socket\") pod \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " Nov 03 18:00:11.594989 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.594789 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-podinfo\") pod \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " Nov 03 18:00:11.594989 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.594821 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-workload-socket\") pod \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " Nov 03 18:00:11.594989 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.594845 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-envoy\") pod \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " Nov 03 18:00:11.594989 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.594873 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-workload-certs\") pod \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\" (UID: \"293f67c8-c8cc-4198-8e03-79c4ae3a710c\") " Nov 03 18:00:11.594989 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.594883 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-data" pod "293f67c8-c8cc-4198-8e03-79c4ae3a710c" (UID: "293f67c8-c8cc-4198-8e03-79c4ae3a710c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:00:11.595315 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.595092 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-credential-socket" pod "293f67c8-c8cc-4198-8e03-79c4ae3a710c" (UID: "293f67c8-c8cc-4198-8e03-79c4ae3a710c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:00:11.595315 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.595189 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-workload-socket" pod "293f67c8-c8cc-4198-8e03-79c4ae3a710c" (UID: "293f67c8-c8cc-4198-8e03-79c4ae3a710c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:00:11.595315 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.595212 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istiod-ca-cert" pod "293f67c8-c8cc-4198-8e03-79c4ae3a710c" (UID: "293f67c8-c8cc-4198-8e03-79c4ae3a710c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:00:11.595315 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.595278 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-workload-certs" pod "293f67c8-c8cc-4198-8e03-79c4ae3a710c" (UID: "293f67c8-c8cc-4198-8e03-79c4ae3a710c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:00:11.595543 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.595346 5805 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-credential-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:00:11.595543 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.595360 5805 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-workload-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:00:11.595543 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.595371 5805 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-data\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:00:11.595543 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.595381 5805 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istiod-ca-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:00:11.597115 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.597086 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-podinfo" pod "293f67c8-c8cc-4198-8e03-79c4ae3a710c" (UID: "293f67c8-c8cc-4198-8e03-79c4ae3a710c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:00:11.597115 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.597099 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-envoy" pod "293f67c8-c8cc-4198-8e03-79c4ae3a710c" (UID: "293f67c8-c8cc-4198-8e03-79c4ae3a710c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:00:11.597243 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.597182 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/293f67c8-c8cc-4198-8e03-79c4ae3a710c-kube-api-access-b6jpf" pod "293f67c8-c8cc-4198-8e03-79c4ae3a710c" (UID: "293f67c8-c8cc-4198-8e03-79c4ae3a710c"). InnerVolumeSpecName "kube-api-access-b6jpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:00:11.597243 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.597189 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-token" pod "293f67c8-c8cc-4198-8e03-79c4ae3a710c" (UID: "293f67c8-c8cc-4198-8e03-79c4ae3a710c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:00:11.696045 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.696013 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:00:11.696045 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.696043 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6jpf\" (UniqueName: \"kubernetes.io/projected/293f67c8-c8cc-4198-8e03-79c4ae3a710c-kube-api-access-b6jpf\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:00:11.696045 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.696053 5805 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-podinfo\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:00:11.696276 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.696063 5805 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-istio-envoy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:00:11.696276 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:11.696072 5805 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/293f67c8-c8cc-4198-8e03-79c4ae3a710c-workload-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:00:12.122929 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:12.122889 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:00:12.212932 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:12.212886 5805 generic.go:356] "Generic (PLEG): container finished" podID="293f67c8-c8cc-4198-8e03-79c4ae3a710c" containerID="9c63ffcc383c8a7996bde478d980c6bb632293d4596dfd5a6272d164b901d810" exitCode=0 Nov 03 18:00:12.213340 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:12.212939 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" Nov 03 18:00:12.213340 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:12.212944 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" event={"ID":"293f67c8-c8cc-4198-8e03-79c4ae3a710c","Type":"ContainerDied","Data":"9c63ffcc383c8a7996bde478d980c6bb632293d4596dfd5a6272d164b901d810"} Nov 03 18:00:12.213340 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:12.212998 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644" event={"ID":"293f67c8-c8cc-4198-8e03-79c4ae3a710c","Type":"ContainerDied","Data":"f369560ad0064258c2d1fa14751b2e3b3d9b96595ab86d4352111826b72fa7db"} Nov 03 18:00:12.213340 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:12.213028 5805 scope.go:122] "RemoveContainer" containerID="9c63ffcc383c8a7996bde478d980c6bb632293d4596dfd5a6272d164b901d810" Nov 03 18:00:12.223353 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:12.223325 5805 scope.go:122] "RemoveContainer" containerID="9c63ffcc383c8a7996bde478d980c6bb632293d4596dfd5a6272d164b901d810" Nov 03 18:00:12.223660 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:00:12.223639 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c63ffcc383c8a7996bde478d980c6bb632293d4596dfd5a6272d164b901d810\": container with ID starting with 9c63ffcc383c8a7996bde478d980c6bb632293d4596dfd5a6272d164b901d810 not found: ID does not exist" containerID="9c63ffcc383c8a7996bde478d980c6bb632293d4596dfd5a6272d164b901d810" Nov 03 18:00:12.223744 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:12.223668 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c63ffcc383c8a7996bde478d980c6bb632293d4596dfd5a6272d164b901d810"} err="failed to get container status \"9c63ffcc383c8a7996bde478d980c6bb632293d4596dfd5a6272d164b901d810\": rpc error: code = NotFound desc = could not find container \"9c63ffcc383c8a7996bde478d980c6bb632293d4596dfd5a6272d164b901d810\": container with ID starting with 9c63ffcc383c8a7996bde478d980c6bb632293d4596dfd5a6272d164b901d810 not found: ID does not exist" Nov 03 18:00:12.414651 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:12.414567 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644"] Nov 03 18:00:12.460941 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:12.460906 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-hn644"] Nov 03 18:00:12.975714 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:12.975682 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="293f67c8-c8cc-4198-8e03-79c4ae3a710c" path="/var/lib/kubelet/pods/293f67c8-c8cc-4198-8e03-79c4ae3a710c/volumes" Nov 03 18:00:16.900119 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:16.900080 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:00:20.969497 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:20.969466 5805 scope.go:122] "RemoveContainer" containerID="050a4916c0b7e15abc6f410df13a73f3fba4e8f5dc0a52d81c2187836470e5e7" Nov 03 18:00:21.268565 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:21.268533 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt" event={"ID":"7ea465a0-b086-41d0-ace1-e47518c0c6c7","Type":"ContainerStarted","Data":"f1e7d211f38291871972301c44d0664f71ddea9e505374d33a42ac35eb473bc1"} Nov 03 18:00:21.403271 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:21.403240 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:00:21.969123 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:21.969095 5805 scope.go:122] "RemoveContainer" containerID="3428cca7c21a9a2ee1ad5fa374089df5fe57aed6ef18e276f5aa4f5c063b8b1a" Nov 03 18:00:22.276229 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:22.276193 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv" event={"ID":"c2477be6-8df0-4e5b-bbd1-3c7fffcc747d","Type":"ContainerStarted","Data":"f68e9e9ca2857331b7ebb22f0808202380be06284cc071412f23c6b99dfbe0bc"} Nov 03 18:00:22.278102 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:22.278078 5805 generic.go:356] "Generic (PLEG): container finished" podID="7ea465a0-b086-41d0-ace1-e47518c0c6c7" containerID="f1e7d211f38291871972301c44d0664f71ddea9e505374d33a42ac35eb473bc1" exitCode=60 Nov 03 18:00:22.278237 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:22.278150 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt" event={"ID":"7ea465a0-b086-41d0-ace1-e47518c0c6c7","Type":"ContainerDied","Data":"f1e7d211f38291871972301c44d0664f71ddea9e505374d33a42ac35eb473bc1"} Nov 03 18:00:22.278237 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:22.278199 5805 scope.go:122] "RemoveContainer" containerID="050a4916c0b7e15abc6f410df13a73f3fba4e8f5dc0a52d81c2187836470e5e7" Nov 03 18:00:22.278535 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:22.278519 5805 scope.go:122] "RemoveContainer" containerID="f1e7d211f38291871972301c44d0664f71ddea9e505374d33a42ac35eb473bc1" Nov 03 18:00:22.278778 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:00:22.278758 5805 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt_odh-ai-gateway-infra(7ea465a0-b086-41d0-ace1-e47518c0c6c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt" podUID="7ea465a0-b086-41d0-ace1-e47518c0c6c7" Nov 03 18:00:22.406854 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:22.406772 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt"] Nov 03 18:00:23.285750 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:23.285719 5805 generic.go:356] "Generic (PLEG): container finished" podID="c2477be6-8df0-4e5b-bbd1-3c7fffcc747d" containerID="f68e9e9ca2857331b7ebb22f0808202380be06284cc071412f23c6b99dfbe0bc" exitCode=60 Nov 03 18:00:23.286245 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:23.285792 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv" event={"ID":"c2477be6-8df0-4e5b-bbd1-3c7fffcc747d","Type":"ContainerDied","Data":"f68e9e9ca2857331b7ebb22f0808202380be06284cc071412f23c6b99dfbe0bc"} Nov 03 18:00:23.286245 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:23.285848 5805 scope.go:122] "RemoveContainer" containerID="3428cca7c21a9a2ee1ad5fa374089df5fe57aed6ef18e276f5aa4f5c063b8b1a" Nov 03 18:00:23.286377 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:23.286260 5805 scope.go:122] "RemoveContainer" containerID="f68e9e9ca2857331b7ebb22f0808202380be06284cc071412f23c6b99dfbe0bc" Nov 03 18:00:23.286600 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:00:23.286579 5805 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv_odh-ai-gateway-infra(c2477be6-8df0-4e5b-bbd1-3c7fffcc747d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv" podUID="c2477be6-8df0-4e5b-bbd1-3c7fffcc747d" Nov 03 18:00:23.411945 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:23.411911 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv"] Nov 03 18:00:23.433710 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:23.433682 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt" Nov 03 18:00:23.619244 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:23.619212 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ea465a0-b086-41d0-ace1-e47518c0c6c7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea465a0-b086-41d0-ace1-e47518c0c6c7-maas-api-service-ca\") pod \"7ea465a0-b086-41d0-ace1-e47518c0c6c7\" (UID: \"7ea465a0-b086-41d0-ace1-e47518c0c6c7\") " Nov 03 18:00:23.619470 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:23.619290 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ea465a0-b086-41d0-ace1-e47518c0c6c7-kube-api-access-t7ntn\" (UniqueName: \"kubernetes.io/projected/7ea465a0-b086-41d0-ace1-e47518c0c6c7-kube-api-access-t7ntn\") pod \"7ea465a0-b086-41d0-ace1-e47518c0c6c7\" (UID: \"7ea465a0-b086-41d0-ace1-e47518c0c6c7\") " Nov 03 18:00:23.619618 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:23.619593 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ea465a0-b086-41d0-ace1-e47518c0c6c7-maas-api-service-ca" pod "7ea465a0-b086-41d0-ace1-e47518c0c6c7" (UID: "7ea465a0-b086-41d0-ace1-e47518c0c6c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:00:23.621611 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:23.621584 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea465a0-b086-41d0-ace1-e47518c0c6c7-kube-api-access-t7ntn" pod "7ea465a0-b086-41d0-ace1-e47518c0c6c7" (UID: "7ea465a0-b086-41d0-ace1-e47518c0c6c7"). InnerVolumeSpecName "kube-api-access-t7ntn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:00:23.721064 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:23.721035 5805 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea465a0-b086-41d0-ace1-e47518c0c6c7-maas-api-service-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:00:23.721064 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:23.721062 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7ntn\" (UniqueName: \"kubernetes.io/projected/7ea465a0-b086-41d0-ace1-e47518c0c6c7-kube-api-access-t7ntn\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:00:24.296025 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:24.295998 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt" Nov 03 18:00:24.296447 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:24.296031 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt" event={"ID":"7ea465a0-b086-41d0-ace1-e47518c0c6c7","Type":"ContainerDied","Data":"7f4833d8111524f34ba4225cc4330bd93ff7c2173e0578eb3e92da2a2ba037fc"} Nov 03 18:00:24.296447 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:24.296123 5805 scope.go:122] "RemoveContainer" containerID="f1e7d211f38291871972301c44d0664f71ddea9e505374d33a42ac35eb473bc1" Nov 03 18:00:24.440886 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:24.440859 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv" Nov 03 18:00:24.508752 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:24.508716 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt"] Nov 03 18:00:24.512683 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:24.512657 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f79c02-29895480-x4slt"] Nov 03 18:00:24.528637 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:24.528604 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2477be6-8df0-4e5b-bbd1-3c7fffcc747d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2477be6-8df0-4e5b-bbd1-3c7fffcc747d-maas-api-service-ca\") pod \"c2477be6-8df0-4e5b-bbd1-3c7fffcc747d\" (UID: \"c2477be6-8df0-4e5b-bbd1-3c7fffcc747d\") " Nov 03 18:00:24.528784 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:24.528654 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2477be6-8df0-4e5b-bbd1-3c7fffcc747d-kube-api-access-ssm5b\" (UniqueName: \"kubernetes.io/projected/c2477be6-8df0-4e5b-bbd1-3c7fffcc747d-kube-api-access-ssm5b\") pod \"c2477be6-8df0-4e5b-bbd1-3c7fffcc747d\" (UID: \"c2477be6-8df0-4e5b-bbd1-3c7fffcc747d\") " Nov 03 18:00:24.528967 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:24.528942 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2477be6-8df0-4e5b-bbd1-3c7fffcc747d-maas-api-service-ca" pod "c2477be6-8df0-4e5b-bbd1-3c7fffcc747d" (UID: "c2477be6-8df0-4e5b-bbd1-3c7fffcc747d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:00:24.529098 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:24.529085 5805 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2477be6-8df0-4e5b-bbd1-3c7fffcc747d-maas-api-service-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:00:24.530925 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:24.530898 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2477be6-8df0-4e5b-bbd1-3c7fffcc747d-kube-api-access-ssm5b" pod "c2477be6-8df0-4e5b-bbd1-3c7fffcc747d" (UID: "c2477be6-8df0-4e5b-bbd1-3c7fffcc747d"). InnerVolumeSpecName "kube-api-access-ssm5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:00:24.629977 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:24.629948 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssm5b\" (UniqueName: \"kubernetes.io/projected/c2477be6-8df0-4e5b-bbd1-3c7fffcc747d-kube-api-access-ssm5b\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:00:24.969858 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:24.969786 5805 scope.go:122] "RemoveContainer" containerID="29e62254df6cb50ffc51cf980a433ab44fa1e69b0e32b7a4495a8ab6741f1b5d" Nov 03 18:00:24.975835 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:24.975810 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ea465a0-b086-41d0-ace1-e47518c0c6c7" path="/var/lib/kubelet/pods/7ea465a0-b086-41d0-ace1-e47518c0c6c7/volumes" Nov 03 18:00:25.302112 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.302082 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv" Nov 03 18:00:25.302112 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.302108 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv" event={"ID":"c2477be6-8df0-4e5b-bbd1-3c7fffcc747d","Type":"ContainerDied","Data":"abfabc199870be44bcd54c6d86bb31926527c077f74e3422fd94334cce81507b"} Nov 03 18:00:25.302713 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.302172 5805 scope.go:122] "RemoveContainer" containerID="f68e9e9ca2857331b7ebb22f0808202380be06284cc071412f23c6b99dfbe0bc" Nov 03 18:00:25.305522 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.305496 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2" event={"ID":"e32b3864-f093-472d-a783-ea1c4f570b80","Type":"ContainerStarted","Data":"d3bfdf82b56113c065ffba82317740d50793b6cb19ff1c61e8906f895426e7ec"} Nov 03 18:00:25.563069 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.563036 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv"] Nov 03 18:00:25.566209 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.566185 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895480-pqchv"] Nov 03 18:00:25.685983 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.685952 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v"] Nov 03 18:00:25.686445 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.686412 5805 cpu_manager.go:414] "containerMap: removing container" podUID="7ea465a0-b086-41d0-ace1-e47518c0c6c7" containerName="cleanup" Nov 03 18:00:25.686445 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.686434 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea465a0-b086-41d0-ace1-e47518c0c6c7" containerName="cleanup" Nov 03 18:00:25.686647 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.686452 5805 cpu_manager.go:414] "containerMap: removing container" podUID="7ea465a0-b086-41d0-ace1-e47518c0c6c7" containerName="cleanup" Nov 03 18:00:25.686647 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.686461 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea465a0-b086-41d0-ace1-e47518c0c6c7" containerName="cleanup" Nov 03 18:00:25.686647 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.686470 5805 cpu_manager.go:414] "containerMap: removing container" podUID="c2477be6-8df0-4e5b-bbd1-3c7fffcc747d" containerName="cleanup" Nov 03 18:00:25.686647 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.686477 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2477be6-8df0-4e5b-bbd1-3c7fffcc747d" containerName="cleanup" Nov 03 18:00:25.686647 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.686528 5805 cpu_manager.go:414] "containerMap: removing container" podUID="c2477be6-8df0-4e5b-bbd1-3c7fffcc747d" containerName="cleanup" Nov 03 18:00:25.686647 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.686536 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2477be6-8df0-4e5b-bbd1-3c7fffcc747d" containerName="cleanup" Nov 03 18:00:25.686647 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.686546 5805 cpu_manager.go:414] "containerMap: removing container" podUID="c2477be6-8df0-4e5b-bbd1-3c7fffcc747d" containerName="cleanup" Nov 03 18:00:25.686647 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.686553 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2477be6-8df0-4e5b-bbd1-3c7fffcc747d" containerName="cleanup" Nov 03 18:00:25.686647 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.686566 5805 cpu_manager.go:414] "containerMap: removing container" podUID="293f67c8-c8cc-4198-8e03-79c4ae3a710c" containerName="istio-proxy" Nov 03 18:00:25.686647 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.686572 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="293f67c8-c8cc-4198-8e03-79c4ae3a710c" containerName="istio-proxy" Nov 03 18:00:25.687121 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.686685 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2477be6-8df0-4e5b-bbd1-3c7fffcc747d" containerName="cleanup" Nov 03 18:00:25.687121 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.686698 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea465a0-b086-41d0-ace1-e47518c0c6c7" containerName="cleanup" Nov 03 18:00:25.687121 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.686706 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2477be6-8df0-4e5b-bbd1-3c7fffcc747d" containerName="cleanup" Nov 03 18:00:25.687121 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.686715 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="293f67c8-c8cc-4198-8e03-79c4ae3a710c" containerName="istio-proxy" Nov 03 18:00:25.687121 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.686728 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea465a0-b086-41d0-ace1-e47518c0c6c7" containerName="cleanup" Nov 03 18:00:25.691415 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.691372 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.740284 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.740262 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-0adcbc-openshift-default-dockercfg-zctsw\"" Nov 03 18:00:25.749537 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.749512 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v"] Nov 03 18:00:25.841604 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.841524 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.841604 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.841557 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.841604 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.841586 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.841827 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.841709 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.841827 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.841761 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8rl6\" (UniqueName: \"kubernetes.io/projected/f403b27f-184b-42fd-839a-92fa3b4cdc2d-kube-api-access-r8rl6\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.841827 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.841792 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.841827 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.841808 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.841982 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.841913 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.841982 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.841957 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.942741 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.942700 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.942901 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.942768 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.942901 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.942815 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.942901 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.942842 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.942901 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.942876 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.943123 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.942930 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.943123 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.943075 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8rl6\" (UniqueName: \"kubernetes.io/projected/f403b27f-184b-42fd-839a-92fa3b4cdc2d-kube-api-access-r8rl6\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.943232 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.943135 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.943232 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.943163 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.943343 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.943250 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.943644 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.943463 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.943644 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.943612 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.943644 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.943614 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.943840 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.943672 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.945108 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.945086 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:25.945307 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:25.945291 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:26.001154 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:26.001125 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:26.002627 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:26.002607 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8rl6\" (UniqueName: \"kubernetes.io/projected/f403b27f-184b-42fd-839a-92fa3b4cdc2d-kube-api-access-r8rl6\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:26.004460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:26.004438 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:26.066163 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:26.063980 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:00:26.235704 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:00:26.235620 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:00:26.313895 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:26.313862 5805 generic.go:356] "Generic (PLEG): container finished" podID="e32b3864-f093-472d-a783-ea1c4f570b80" containerID="d3bfdf82b56113c065ffba82317740d50793b6cb19ff1c61e8906f895426e7ec" exitCode=60 Nov 03 18:00:26.314371 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:26.313905 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2" event={"ID":"e32b3864-f093-472d-a783-ea1c4f570b80","Type":"ContainerDied","Data":"d3bfdf82b56113c065ffba82317740d50793b6cb19ff1c61e8906f895426e7ec"} Nov 03 18:00:26.314371 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:26.313957 5805 scope.go:122] "RemoveContainer" containerID="29e62254df6cb50ffc51cf980a433ab44fa1e69b0e32b7a4495a8ab6741f1b5d" Nov 03 18:00:26.314371 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:26.314346 5805 scope.go:122] "RemoveContainer" containerID="d3bfdf82b56113c065ffba82317740d50793b6cb19ff1c61e8906f895426e7ec" Nov 03 18:00:26.314669 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:00:26.314644 5805 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2_odh-ai-gateway-infra(e32b3864-f093-472d-a783-ea1c4f570b80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2" podUID="e32b3864-f093-472d-a783-ea1c4f570b80" Nov 03 18:00:26.431647 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:26.431615 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v"] Nov 03 18:00:26.437650 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:00:26.437615 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf403b27f_184b_42fd_839a_92fa3b4cdc2d.slice/crio-97185dcbccd8b63a79ead35e6025ad50c10db052ed61f2be1f4cc1ff27f5c2b2 WatchSource:0}: Error finding container 97185dcbccd8b63a79ead35e6025ad50c10db052ed61f2be1f4cc1ff27f5c2b2: Status 404 returned error can't find the container with id 97185dcbccd8b63a79ead35e6025ad50c10db052ed61f2be1f4cc1ff27f5c2b2 Nov 03 18:00:26.439945 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:26.439914 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:00:26.440032 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:26.439975 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:00:26.440032 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:26.440000 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:00:26.472615 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:26.472591 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2"] Nov 03 18:00:26.976052 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:26.976015 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2477be6-8df0-4e5b-bbd1-3c7fffcc747d" path="/var/lib/kubelet/pods/c2477be6-8df0-4e5b-bbd1-3c7fffcc747d/volumes" Nov 03 18:00:27.350418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:27.346644 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" event={"ID":"f403b27f-184b-42fd-839a-92fa3b4cdc2d","Type":"ContainerStarted","Data":"a1ef37a20a86aeb6b14d713c064c54d9c5e4b4dd435cf111e33341c312ecce7c"} Nov 03 18:00:27.350418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:27.346714 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" event={"ID":"f403b27f-184b-42fd-839a-92fa3b4cdc2d","Type":"ContainerStarted","Data":"97185dcbccd8b63a79ead35e6025ad50c10db052ed61f2be1f4cc1ff27f5c2b2"} Nov 03 18:00:27.486491 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:27.486464 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2" Nov 03 18:00:27.565293 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:27.565235 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" podStartSLOduration=2.565221142 podStartE2EDuration="2.565221142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:27.565132788 +0000 UTC m=+864.698761819" watchObservedRunningTime="2026-11-03 18:00:27.565221142 +0000 UTC m=+864.698850152" Nov 03 18:00:27.659686 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:27.659598 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e32b3864-f093-472d-a783-ea1c4f570b80-kube-api-access-p9nvc\" (UniqueName: \"kubernetes.io/projected/e32b3864-f093-472d-a783-ea1c4f570b80-kube-api-access-p9nvc\") pod \"e32b3864-f093-472d-a783-ea1c4f570b80\" (UID: \"e32b3864-f093-472d-a783-ea1c4f570b80\") " Nov 03 18:00:27.659825 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:27.659749 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e32b3864-f093-472d-a783-ea1c4f570b80-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e32b3864-f093-472d-a783-ea1c4f570b80-maas-api-service-ca\") pod \"e32b3864-f093-472d-a783-ea1c4f570b80\" (UID: \"e32b3864-f093-472d-a783-ea1c4f570b80\") " Nov 03 18:00:27.660103 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:27.660072 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e32b3864-f093-472d-a783-ea1c4f570b80-maas-api-service-ca" pod "e32b3864-f093-472d-a783-ea1c4f570b80" (UID: "e32b3864-f093-472d-a783-ea1c4f570b80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:00:27.661821 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:27.661796 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32b3864-f093-472d-a783-ea1c4f570b80-kube-api-access-p9nvc" pod "e32b3864-f093-472d-a783-ea1c4f570b80" (UID: "e32b3864-f093-472d-a783-ea1c4f570b80"). InnerVolumeSpecName "kube-api-access-p9nvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:00:27.760597 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:27.760560 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9nvc\" (UniqueName: \"kubernetes.io/projected/e32b3864-f093-472d-a783-ea1c4f570b80-kube-api-access-p9nvc\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:00:27.760597 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:27.760594 5805 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e32b3864-f093-472d-a783-ea1c4f570b80-maas-api-service-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:00:28.004690 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:28.004607 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:28.006016 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:28.005989 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 18:00:28.006117 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:28.006037 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" podUID="f403b27f-184b-42fd-839a-92fa3b4cdc2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 18:00:28.353918 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:28.353890 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2" Nov 03 18:00:28.354368 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:28.353931 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2" event={"ID":"e32b3864-f093-472d-a783-ea1c4f570b80","Type":"ContainerDied","Data":"9fc65c62b89c8eca0c17ac235b1d87054ad388cfb0ef1708ee06230a754d10d7"} Nov 03 18:00:28.354368 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:28.353995 5805 scope.go:122] "RemoveContainer" containerID="d3bfdf82b56113c065ffba82317740d50793b6cb19ff1c61e8906f895426e7ec" Nov 03 18:00:28.512912 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:28.512877 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2"] Nov 03 18:00:28.517243 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:28.517217 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895480-9pnk2"] Nov 03 18:00:28.976332 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:28.976298 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e32b3864-f093-472d-a783-ea1c4f570b80" path="/var/lib/kubelet/pods/e32b3864-f093-472d-a783-ea1c4f570b80/volumes" Nov 03 18:00:29.005643 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:29.005615 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 18:00:29.005784 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:29.005661 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" podUID="f403b27f-184b-42fd-839a-92fa3b4cdc2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 18:00:30.005106 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:30.005071 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 18:00:30.005501 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:30.005121 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" podUID="f403b27f-184b-42fd-839a-92fa3b4cdc2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 18:00:30.778453 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:30.778418 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:00:31.004907 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:31.004879 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 18:00:31.005072 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:31.004927 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" podUID="f403b27f-184b-42fd-839a-92fa3b4cdc2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 18:00:32.005456 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:32.005426 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 18:00:32.005843 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:32.005472 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" podUID="f403b27f-184b-42fd-839a-92fa3b4cdc2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 18:00:33.234101 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:33.234008 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:33.234625 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:33.234513 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:33.235356 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:33.235335 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:00:35.376343 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:35.376303 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:00:39.977505 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:39.977474 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:00:44.584309 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:44.584271 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:00:49.164041 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:49.164003 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:00:53.695744 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:53.695706 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:00:56.235041 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:00:56.235003 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:00:58.650750 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:00:58.650700 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:01:03.124282 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:03.124257 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 18:01:03.124690 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:03.124574 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 18:01:03.378383 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:03.378308 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:01:08.404132 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:08.404088 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:01:09.445542 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.445510 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5"] Nov 03 18:01:09.445968 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.445955 5805 cpu_manager.go:414] "containerMap: removing container" podUID="e32b3864-f093-472d-a783-ea1c4f570b80" containerName="cleanup" Nov 03 18:01:09.446015 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.445969 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e32b3864-f093-472d-a783-ea1c4f570b80" containerName="cleanup" Nov 03 18:01:09.446015 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.445978 5805 cpu_manager.go:414] "containerMap: removing container" podUID="7ea465a0-b086-41d0-ace1-e47518c0c6c7" containerName="cleanup" Nov 03 18:01:09.446015 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.445983 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea465a0-b086-41d0-ace1-e47518c0c6c7" containerName="cleanup" Nov 03 18:01:09.446015 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.445999 5805 cpu_manager.go:414] "containerMap: removing container" podUID="e32b3864-f093-472d-a783-ea1c4f570b80" containerName="cleanup" Nov 03 18:01:09.446015 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.446004 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e32b3864-f093-472d-a783-ea1c4f570b80" containerName="cleanup" Nov 03 18:01:09.446181 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.446070 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2477be6-8df0-4e5b-bbd1-3c7fffcc747d" containerName="cleanup" Nov 03 18:01:09.446181 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.446081 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea465a0-b086-41d0-ace1-e47518c0c6c7" containerName="cleanup" Nov 03 18:01:09.446181 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.446087 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="e32b3864-f093-472d-a783-ea1c4f570b80" containerName="cleanup" Nov 03 18:01:09.446181 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.446093 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="e32b3864-f093-472d-a783-ea1c4f570b80" containerName="cleanup" Nov 03 18:01:09.446181 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.446101 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="e32b3864-f093-472d-a783-ea1c4f570b80" containerName="cleanup" Nov 03 18:01:09.449347 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.449326 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:01:09.479480 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.479446 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-tmp-dir\") pod \"distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5\" (UID: \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:01:09.479614 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.479500 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-model-cache\") pod \"distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5\" (UID: \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:01:09.479655 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.479627 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce3c87ae-e083-48dd-b3f3-4d18b917c395-tls-certs\") pod \"distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5\" (UID: \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:01:09.479710 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.479694 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-home\") pod \"distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5\" (UID: \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:01:09.479813 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.479793 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-dshm\") pod \"distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5\" (UID: \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:01:09.479869 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.479847 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9vgb\" (UniqueName: \"kubernetes.io/projected/ce3c87ae-e083-48dd-b3f3-4d18b917c395-kube-api-access-j9vgb\") pod \"distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5\" (UID: \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:01:09.547793 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.547765 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2b4f46\"/\"openshift-service-ca.crt\"" Nov 03 18:01:09.550383 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.550347 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2b4f46\"/\"default-dockercfg-hgn5n\"" Nov 03 18:01:09.550522 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.550462 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2b4f46\"/\"kube-root-ca.crt\"" Nov 03 18:01:09.550522 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.550508 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2b4f46\"/\"distinct-w5-2b4f46-kserve-self-signed-certs\"" Nov 03 18:01:09.553217 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.553197 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5"] Nov 03 18:01:09.581245 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.581213 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9vgb\" (UniqueName: \"kubernetes.io/projected/ce3c87ae-e083-48dd-b3f3-4d18b917c395-kube-api-access-j9vgb\") pod \"distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5\" (UID: \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:01:09.581431 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.581329 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-tmp-dir\") pod \"distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5\" (UID: \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:01:09.581431 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.581367 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-model-cache\") pod \"distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5\" (UID: \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:01:09.581566 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.581442 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce3c87ae-e083-48dd-b3f3-4d18b917c395-tls-certs\") pod \"distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5\" (UID: \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:01:09.581566 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.581496 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-home\") pod \"distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5\" (UID: \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:01:09.581566 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.581535 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-dshm\") pod \"distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5\" (UID: \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:01:09.582081 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.582054 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-tmp-dir\") pod \"distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5\" (UID: \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:01:09.582194 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.582059 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-model-cache\") pod \"distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5\" (UID: \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:01:09.582194 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.582167 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-home\") pod \"distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5\" (UID: \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:01:09.584441 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.584418 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-dshm\") pod \"distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5\" (UID: \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:01:09.585533 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.585510 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce3c87ae-e083-48dd-b3f3-4d18b917c395-tls-certs\") pod \"distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5\" (UID: \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:01:09.600345 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.600320 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:01:09.640188 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.640155 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9vgb\" (UniqueName: \"kubernetes.io/projected/ce3c87ae-e083-48dd-b3f3-4d18b917c395-kube-api-access-j9vgb\") pod \"distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5\" (UID: \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\") " pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:01:09.761629 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:09.761546 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:01:10.146831 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:10.146794 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:01:10.172727 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:10.172693 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5"] Nov 03 18:01:10.178873 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:01:10.178843 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce3c87ae_e083_48dd_b3f3_4d18b917c395.slice/crio-ef9783f9870367d4425b866551b974635660074680b7d13f4745783c883a136b WatchSource:0}: Error finding container ef9783f9870367d4425b866551b974635660074680b7d13f4745783c883a136b: Status 404 returned error can't find the container with id ef9783f9870367d4425b866551b974635660074680b7d13f4745783c883a136b Nov 03 18:01:10.382741 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:10.382711 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:01:10.582914 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:10.582882 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" event={"ID":"ce3c87ae-e083-48dd-b3f3-4d18b917c395","Type":"ContainerStarted","Data":"2df83bd5bb7e52464d02cf50f16ddc1c2feac2bee8133bfab44fe7e43c511b5b"} Nov 03 18:01:10.583314 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:10.582927 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" event={"ID":"ce3c87ae-e083-48dd-b3f3-4d18b917c395","Type":"ContainerStarted","Data":"ef9783f9870367d4425b866551b974635660074680b7d13f4745783c883a136b"} Nov 03 18:01:10.695140 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:10.695086 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" podStartSLOduration=1.695071274 podStartE2EDuration="1.695071274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:10.693093832 +0000 UTC m=+907.826722853" watchObservedRunningTime="2026-11-03 18:01:10.695071274 +0000 UTC m=+907.828700284" Nov 03 18:01:10.762159 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:10.762125 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:01:12.973368 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:12.973279 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:01:17.615494 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:17.615458 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:01:19.762318 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:19.762280 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:01:19.774816 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:19.774791 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:01:20.650148 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:20.650116 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:01:21.963736 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:21.963704 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z"] Nov 03 18:01:21.964370 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:21.964343 5805 cpu_manager.go:414] "containerMap: removing container" podUID="e32b3864-f093-472d-a783-ea1c4f570b80" containerName="cleanup" Nov 03 18:01:21.964370 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:21.964368 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e32b3864-f093-472d-a783-ea1c4f570b80" containerName="cleanup" Nov 03 18:01:21.967018 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:21.967000 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.018472 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.018439 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.018620 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.018538 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.018620 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.018591 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kgwj\" (UniqueName: \"kubernetes.io/projected/f8c9e705-3831-499a-9ff4-3b03aeac8617-kube-api-access-8kgwj\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.018717 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.018619 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.018818 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.018799 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.018876 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.018860 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.018937 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.018917 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.019008 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.018983 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c9e705-3831-499a-9ff4-3b03aeac8617-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.019078 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.019063 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.089715 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.089682 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z"] Nov 03 18:01:22.120362 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.120329 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.120541 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.120370 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.120541 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.120386 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c9e705-3831-499a-9ff4-3b03aeac8617-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.120541 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.120465 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.120541 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.120507 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.120770 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.120558 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.120770 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.120703 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kgwj\" (UniqueName: \"kubernetes.io/projected/f8c9e705-3831-499a-9ff4-3b03aeac8617-kube-api-access-8kgwj\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.120770 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.120745 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.121119 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.120929 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.121119 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.120993 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.121119 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.121009 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.121119 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.121068 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.121326 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.121306 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c9e705-3831-499a-9ff4-3b03aeac8617-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.121447 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.121431 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.123328 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.123302 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.123481 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.123466 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.179087 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.179047 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.179416 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.179381 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kgwj\" (UniqueName: \"kubernetes.io/projected/f8c9e705-3831-499a-9ff4-3b03aeac8617-kube-api-access-8kgwj\") pod \"maas-default-gateway-openshift-default-6dcf757899-zjg5z\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.279497 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.279386 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:22.309861 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.309828 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:01:22.715206 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.715174 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z"] Nov 03 18:01:22.721088 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:01:22.721065 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c9e705_3831_499a_9ff4_3b03aeac8617.slice/crio-7c36578acfe3a9861309eab6896d8c05283d3f2938b0399d6328235f0900ebce WatchSource:0}: Error finding container 7c36578acfe3a9861309eab6896d8c05283d3f2938b0399d6328235f0900ebce: Status 404 returned error can't find the container with id 7c36578acfe3a9861309eab6896d8c05283d3f2938b0399d6328235f0900ebce Nov 03 18:01:22.723496 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.723458 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:01:22.723578 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.723518 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:01:22.723578 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:22.723545 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:01:23.659754 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:23.659714 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" event={"ID":"f8c9e705-3831-499a-9ff4-3b03aeac8617","Type":"ContainerStarted","Data":"1089f0d13fee4c34da914ac3ae10a87ff4d9c039e02dbd63cbf50aefc6f544db"} Nov 03 18:01:23.660253 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:23.659770 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" event={"ID":"f8c9e705-3831-499a-9ff4-3b03aeac8617","Type":"ContainerStarted","Data":"7c36578acfe3a9861309eab6896d8c05283d3f2938b0399d6328235f0900ebce"} Nov 03 18:01:23.866968 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:23.866912 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" podStartSLOduration=2.866896807 podStartE2EDuration="2.866896807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:23.833723823 +0000 UTC m=+920.967352859" watchObservedRunningTime="2026-11-03 18:01:23.866896807 +0000 UTC m=+921.000525816" Nov 03 18:01:24.280263 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:24.280222 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:24.281902 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:24.281876 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-zjg5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 18:01:24.282075 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:24.281927 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" podUID="f8c9e705-3831-499a-9ff4-3b03aeac8617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 18:01:25.280247 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:25.280214 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-zjg5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 18:01:25.280776 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:25.280269 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" podUID="f8c9e705-3831-499a-9ff4-3b03aeac8617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 18:01:26.262689 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:01:26.262647 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:01:26.280283 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:26.280254 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-zjg5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 18:01:26.280663 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:26.280307 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" podUID="f8c9e705-3831-499a-9ff4-3b03aeac8617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 18:01:27.044418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:27.034453 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:01:27.279896 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:27.279862 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-zjg5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 18:01:27.280070 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:27.279920 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" podUID="f8c9e705-3831-499a-9ff4-3b03aeac8617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 18:01:28.280541 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:28.280505 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-zjg5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 18:01:28.280930 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:28.280561 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" podUID="f8c9e705-3831-499a-9ff4-3b03aeac8617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 18:01:29.280599 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:29.280556 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-zjg5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 18:01:29.280979 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:29.280622 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" podUID="f8c9e705-3831-499a-9ff4-3b03aeac8617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 18:01:31.114250 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:31.114207 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:31.114691 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:31.114656 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:31.115481 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:31.115460 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:01:31.691212 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:31.691180 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:01:32.823479 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:32.823445 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:01:36.266806 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:36.266776 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:01:39.566235 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.566202 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd"] Nov 03 18:01:39.569457 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.569430 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.618940 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.618917 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-f79c02-openshift-default-dockercfg-qxdds\"" Nov 03 18:01:39.673312 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.673283 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd"] Nov 03 18:01:39.714354 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.714331 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86ead782-7e42-4a09-854e-e5e3aa485554-istio-token\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.714505 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.714363 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-credential-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.714505 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.714480 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ead782-7e42-4a09-854e-e5e3aa485554-istio-podinfo\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.714592 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.714542 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-workload-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.714592 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.714575 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pjzm\" (UniqueName: \"kubernetes.io/projected/86ead782-7e42-4a09-854e-e5e3aa485554-kube-api-access-5pjzm\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.714692 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.714601 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-workload-certs\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.714692 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.714685 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ead782-7e42-4a09-854e-e5e3aa485554-istiod-ca-cert\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.714774 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.714705 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-istio-data\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.714774 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.714723 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-istio-envoy\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.815651 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.815622 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ead782-7e42-4a09-854e-e5e3aa485554-istio-podinfo\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.815782 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.815659 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-workload-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.815782 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.815688 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pjzm\" (UniqueName: \"kubernetes.io/projected/86ead782-7e42-4a09-854e-e5e3aa485554-kube-api-access-5pjzm\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.815782 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.815713 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-workload-certs\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.815782 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.815743 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ead782-7e42-4a09-854e-e5e3aa485554-istiod-ca-cert\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.815782 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.815762 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-istio-data\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.816031 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.815876 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-istio-envoy\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.816031 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.815970 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86ead782-7e42-4a09-854e-e5e3aa485554-istio-token\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.816031 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.816006 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-credential-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.816180 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.816075 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-workload-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.816180 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.816147 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-istio-data\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.816339 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.816292 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-workload-certs\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.816430 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.816388 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-credential-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.816605 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.816581 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ead782-7e42-4a09-854e-e5e3aa485554-istiod-ca-cert\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.818117 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.818095 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-istio-envoy\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.818217 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.818190 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ead782-7e42-4a09-854e-e5e3aa485554-istio-podinfo\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.873952 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.873923 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86ead782-7e42-4a09-854e-e5e3aa485554-istio-token\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.874096 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.874072 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pjzm\" (UniqueName: \"kubernetes.io/projected/86ead782-7e42-4a09-854e-e5e3aa485554-kube-api-access-5pjzm\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-957dd\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:39.882446 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:39.882426 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:40.319685 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:40.319655 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd"] Nov 03 18:01:40.330327 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:40.330294 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:01:40.330468 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:40.330366 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:01:40.330468 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:40.330415 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:01:40.755638 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:40.755538 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" event={"ID":"86ead782-7e42-4a09-854e-e5e3aa485554","Type":"ContainerStarted","Data":"8466544c8d87f41b83fbc8a4ff089ded9351c944f132e03ab14111ff2472b1ab"} Nov 03 18:01:40.755638 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:40.755607 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" event={"ID":"86ead782-7e42-4a09-854e-e5e3aa485554","Type":"ContainerStarted","Data":"12e8fbbf4695f6b1eee2a4afea659e5b90054d36e3b72fe714cf017c88c8004e"} Nov 03 18:01:40.877367 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:40.877313 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" podStartSLOduration=1.877294852 podStartE2EDuration="1.877294852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:40.876689698 +0000 UTC m=+938.010318723" watchObservedRunningTime="2026-11-03 18:01:40.877294852 +0000 UTC m=+938.010923861" Nov 03 18:01:40.882868 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:40.882842 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:40.886115 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:40.886082 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:01:41.884788 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:41.884756 5805 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 18:01:41.885179 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:41.884809 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" podUID="86ead782-7e42-4a09-854e-e5e3aa485554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 18:01:42.883257 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:42.883222 5805 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 18:01:42.883451 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:42.883273 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" podUID="86ead782-7e42-4a09-854e-e5e3aa485554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 18:01:43.883571 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:43.883532 5805 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 18:01:43.883972 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:43.883586 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" podUID="86ead782-7e42-4a09-854e-e5e3aa485554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 18:01:44.883872 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:44.883831 5805 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 18:01:44.884276 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:44.883898 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" podUID="86ead782-7e42-4a09-854e-e5e3aa485554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 18:01:45.634871 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:45.634799 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:01:45.882830 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:45.882795 5805 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 03 18:01:45.882983 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:45.882850 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" podUID="86ead782-7e42-4a09-854e-e5e3aa485554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 03 18:01:47.098681 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:47.098646 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:47.099190 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:47.098951 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:47.099849 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:47.099827 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:01:50.274512 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:50.274476 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:01:54.896321 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:54.896285 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:01:54.954309 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:54.954281 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn"] Nov 03 18:01:54.957770 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:54.957750 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.007466 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.007436 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-ef9633-openshift-default-dockercfg-49fqv\"" Nov 03 18:01:55.056143 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.056110 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn"] Nov 03 18:01:55.086896 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.086871 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.087041 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.086921 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.087041 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.086943 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.087118 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.087075 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.087163 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.087117 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dc4m\" (UniqueName: \"kubernetes.io/projected/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-kube-api-access-6dc4m\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.087253 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.087231 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.087327 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.087311 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.087419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.087379 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.087461 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.087443 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.188323 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.188260 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.188323 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.188310 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.188548 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.188341 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.188548 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.188424 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.188548 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.188480 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.188548 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.188505 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.188548 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.188544 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.188819 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.188663 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dc4m\" (UniqueName: \"kubernetes.io/projected/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-kube-api-access-6dc4m\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.188819 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.188705 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.188819 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.188763 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.188968 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.188903 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.189027 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.189004 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.189148 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.189126 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.189271 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.189253 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.190794 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.190771 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.190875 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.190815 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.246232 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.246203 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.246385 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.246366 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dc4m\" (UniqueName: \"kubernetes.io/projected/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-kube-api-access-6dc4m\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.270340 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.270314 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:55.280875 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.280751 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x"] Nov 03 18:01:55.283748 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.283732 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.346527 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.346503 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-2b4f46-openshift-default-dockercfg-5qdjt\"" Nov 03 18:01:55.390726 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.390696 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.390888 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.390733 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.390888 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.390863 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10efc538-5a7a-40ce-b6be-819a2cddbec3-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.391007 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.390889 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.391007 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.390908 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.391007 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.390944 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.391221 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.391052 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.391221 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.391101 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.391221 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.391194 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv5bp\" (UniqueName: \"kubernetes.io/projected/10efc538-5a7a-40ce-b6be-819a2cddbec3-kube-api-access-wv5bp\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.406846 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.406775 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x"] Nov 03 18:01:55.491970 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.491879 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv5bp\" (UniqueName: \"kubernetes.io/projected/10efc538-5a7a-40ce-b6be-819a2cddbec3-kube-api-access-wv5bp\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.491970 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.491931 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.491970 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.491957 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.492255 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.492062 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10efc538-5a7a-40ce-b6be-819a2cddbec3-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.492255 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.492102 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.492255 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.492128 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.492255 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.492209 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.492503 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.492268 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.492503 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.492321 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.492613 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.492574 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.492670 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.492609 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.492713 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.492694 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.492788 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.492772 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.493021 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.493001 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10efc538-5a7a-40ce-b6be-819a2cddbec3-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.494280 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.494249 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.494637 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.494619 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.549559 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.549539 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv5bp\" (UniqueName: \"kubernetes.io/projected/10efc538-5a7a-40ce-b6be-819a2cddbec3-kube-api-access-wv5bp\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.552326 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.552302 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.601029 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.601005 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:55.701208 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.701179 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn"] Nov 03 18:01:55.707732 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:01:55.707697 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf745c4ba_4f19_485f_9c87_3f5f2ab6ecb4.slice/crio-59a06c829e54835d8112d13cfc7898cea0160b2024f0b718913ce1fedba2eeb9 WatchSource:0}: Error finding container 59a06c829e54835d8112d13cfc7898cea0160b2024f0b718913ce1fedba2eeb9: Status 404 returned error can't find the container with id 59a06c829e54835d8112d13cfc7898cea0160b2024f0b718913ce1fedba2eeb9 Nov 03 18:01:55.710727 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.710690 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:01:55.710824 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.710757 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:01:55.710824 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.710783 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:01:55.838018 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.837982 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" event={"ID":"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4","Type":"ContainerStarted","Data":"97755c9f8d76b8f059d8dc6f19b2800e415153b4e33f89ae39dfddf0cd9202f1"} Nov 03 18:01:55.838202 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.838051 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" event={"ID":"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4","Type":"ContainerStarted","Data":"59a06c829e54835d8112d13cfc7898cea0160b2024f0b718913ce1fedba2eeb9"} Nov 03 18:01:55.955224 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.955167 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" podStartSLOduration=1.955154015 podStartE2EDuration="1.955154015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:55.954628874 +0000 UTC m=+953.088257913" watchObservedRunningTime="2026-11-03 18:01:55.955154015 +0000 UTC m=+953.088783025" Nov 03 18:01:55.977858 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.977829 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x"] Nov 03 18:01:55.984501 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:01:55.984472 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10efc538_5a7a_40ce_b6be_819a2cddbec3.slice/crio-cd96d2f32030bb7fada9156a1b8e75be6e837345fb4aee762812204a3b70eb30 WatchSource:0}: Error finding container cd96d2f32030bb7fada9156a1b8e75be6e837345fb4aee762812204a3b70eb30: Status 404 returned error can't find the container with id cd96d2f32030bb7fada9156a1b8e75be6e837345fb4aee762812204a3b70eb30 Nov 03 18:01:55.987005 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.986973 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:01:55.987093 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.987036 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:01:55.987093 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:55.987069 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:01:56.256123 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:01:56.256027 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:01:56.270963 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:56.270891 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:01:56.272749 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:56.272685 5805 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 18:01:56.272749 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:56.272740 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" podUID="f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 18:01:56.931420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:56.930445 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" event={"ID":"10efc538-5a7a-40ce-b6be-819a2cddbec3","Type":"ContainerStarted","Data":"3fecda1bdcbd492a83a22186e1898a033530916f0b4ba8f14f1ffe7003a3d4d2"} Nov 03 18:01:56.931420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:56.930537 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" event={"ID":"10efc538-5a7a-40ce-b6be-819a2cddbec3","Type":"ContainerStarted","Data":"cd96d2f32030bb7fada9156a1b8e75be6e837345fb4aee762812204a3b70eb30"} Nov 03 18:01:57.112419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:57.111409 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" podStartSLOduration=2.111374909 podStartE2EDuration="2.111374909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:57.097262164 +0000 UTC m=+954.230891175" watchObservedRunningTime="2026-11-03 18:01:57.111374909 +0000 UTC m=+954.245003923" Nov 03 18:01:57.276473 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:57.272784 5805 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 18:01:57.276473 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:57.272849 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" podUID="f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 18:01:57.601830 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:57.601795 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:01:57.603413 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:57.603375 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 18:01:57.603606 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:57.603585 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" podUID="10efc538-5a7a-40ce-b6be-819a2cddbec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 18:01:58.271529 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:58.271013 5805 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 18:01:58.271529 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:58.271078 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" podUID="f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 18:01:58.602104 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:58.602076 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 18:01:58.602351 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:58.602327 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" podUID="10efc538-5a7a-40ce-b6be-819a2cddbec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 18:01:59.271414 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:59.270796 5805 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 18:01:59.271414 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:59.270859 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" podUID="f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 18:01:59.602028 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:59.601992 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 18:01:59.602183 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:59.602041 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" podUID="10efc538-5a7a-40ce-b6be-819a2cddbec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 18:01:59.768774 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:01:59.764740 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:00.271514 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:00.270808 5805 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 18:02:00.271514 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:00.270874 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" podUID="f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 18:02:00.602018 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:00.601984 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 18:02:00.602185 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:00.602041 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" podUID="10efc538-5a7a-40ce-b6be-819a2cddbec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 18:02:01.277598 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:01.277566 5805 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Nov 03 18:02:01.278164 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:01.278131 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" podUID="f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Nov 03 18:02:01.310973 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:01.310891 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:01.603448 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:01.601359 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 18:02:01.603448 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:01.601434 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" podUID="10efc538-5a7a-40ce-b6be-819a2cddbec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 18:02:02.545074 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:02.545045 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:02:02.545535 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:02.545422 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:02:02.546388 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:02.546359 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:02:02.601571 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:02.601542 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Nov 03 18:02:02.601739 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:02.601600 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" podUID="10efc538-5a7a-40ce-b6be-819a2cddbec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Nov 03 18:02:03.606252 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:03.606219 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:02:03.606657 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:03.606492 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:02:03.607353 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:03.607328 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:02:04.575438 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:04.575375 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:09.189852 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:09.189810 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:10.490885 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:10.490851 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:13.810282 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:13.810216 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:14.444035 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:14.443999 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:18.487094 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:18.487060 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:22.991901 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:22.991864 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:25.310122 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:25.310086 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52"] Nov 03 18:02:25.313297 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:25.313273 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:02:25.408977 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:25.408942 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-0ccbb9\"/\"distinct-w0-0ccbb9-kserve-self-signed-certs\"" Nov 03 18:02:25.411364 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:25.411343 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52"] Nov 03 18:02:25.470888 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:25.470853 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:25.488268 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:25.488229 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlfm2\" (UniqueName: \"kubernetes.io/projected/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-kube-api-access-nlfm2\") pod \"distinct-w0-0ccbb9-kserve-5589bf4d88-82b52\" (UID: \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:02:25.488484 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:25.488341 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-dshm\") pod \"distinct-w0-0ccbb9-kserve-5589bf4d88-82b52\" (UID: \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:02:25.488484 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:25.488414 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-tmp-dir\") pod \"distinct-w0-0ccbb9-kserve-5589bf4d88-82b52\" (UID: \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:02:25.488594 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:25.488488 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-home\") pod \"distinct-w0-0ccbb9-kserve-5589bf4d88-82b52\" (UID: \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:02:25.488594 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:25.488509 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-model-cache\") pod \"distinct-w0-0ccbb9-kserve-5589bf4d88-82b52\" (UID: \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:02:25.488685 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:25.488617 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-tls-certs\") pod \"distinct-w0-0ccbb9-kserve-5589bf4d88-82b52\" (UID: \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:02:25.590184 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:25.590092 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-tmp-dir\") pod \"distinct-w0-0ccbb9-kserve-5589bf4d88-82b52\" (UID: \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:02:25.590184 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:25.590160 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-home\") pod \"distinct-w0-0ccbb9-kserve-5589bf4d88-82b52\" (UID: \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:02:25.590390 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:25.590187 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-model-cache\") pod \"distinct-w0-0ccbb9-kserve-5589bf4d88-82b52\" (UID: \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:02:25.590390 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:25.590273 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-tls-certs\") pod \"distinct-w0-0ccbb9-kserve-5589bf4d88-82b52\" (UID: \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:02:25.590510 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:25.590412 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlfm2\" (UniqueName: \"kubernetes.io/projected/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-kube-api-access-nlfm2\") pod \"distinct-w0-0ccbb9-kserve-5589bf4d88-82b52\" (UID: \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:02:25.590510 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:25.590480 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-dshm\") pod \"distinct-w0-0ccbb9-kserve-5589bf4d88-82b52\" (UID: \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:02:25.590624 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:25.590562 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-tmp-dir\") pod \"distinct-w0-0ccbb9-kserve-5589bf4d88-82b52\" (UID: \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:02:25.590685 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:25.590627 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-home\") pod \"distinct-w0-0ccbb9-kserve-5589bf4d88-82b52\" (UID: \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:02:25.590685 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:25.590641 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-model-cache\") pod \"distinct-w0-0ccbb9-kserve-5589bf4d88-82b52\" (UID: \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:02:25.592803 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:25.592776 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-dshm\") pod \"distinct-w0-0ccbb9-kserve-5589bf4d88-82b52\" (UID: \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:02:25.593105 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:25.593080 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-tls-certs\") pod \"distinct-w0-0ccbb9-kserve-5589bf4d88-82b52\" (UID: \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:02:25.648740 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:25.648713 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlfm2\" (UniqueName: \"kubernetes.io/projected/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-kube-api-access-nlfm2\") pod \"distinct-w0-0ccbb9-kserve-5589bf4d88-82b52\" (UID: \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\") " pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:02:25.924418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:25.924309 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:02:26.236012 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:02:26.235930 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:02:26.299143 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:26.299116 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52"] Nov 03 18:02:26.304885 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:02:26.304860 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2353fc1e_3b17_41f1_a8f4_47c2bc2cc260.slice/crio-c6011e3c9888333a7e6062386807e928036b75d07b7a0545200af1d650f9c0eb WatchSource:0}: Error finding container c6011e3c9888333a7e6062386807e928036b75d07b7a0545200af1d650f9c0eb: Status 404 returned error can't find the container with id c6011e3c9888333a7e6062386807e928036b75d07b7a0545200af1d650f9c0eb Nov 03 18:02:27.047695 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.047663 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t"] Nov 03 18:02:27.051213 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.051194 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.100845 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.100819 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-a86228-openshift-default-dockercfg-pqbnk\"" Nov 03 18:02:27.137707 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.137674 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" event={"ID":"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260","Type":"ContainerStarted","Data":"b639272e70bb94c33b737d2f51c99472cd6928386466bd3dd69ddbc557ecbc66"} Nov 03 18:02:27.137837 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.137725 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" event={"ID":"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260","Type":"ContainerStarted","Data":"c6011e3c9888333a7e6062386807e928036b75d07b7a0545200af1d650f9c0eb"} Nov 03 18:02:27.153368 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.153338 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t"] Nov 03 18:02:27.208774 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.208741 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4120351-f702-437b-aa9b-40979fa30455-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.208960 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.208873 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.208960 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.208899 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4120351-f702-437b-aa9b-40979fa30455-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.208960 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.208945 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.209085 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.209054 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4120351-f702-437b-aa9b-40979fa30455-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.209128 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.209096 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.209128 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.209116 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.209254 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.209236 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbsll\" (UniqueName: \"kubernetes.io/projected/e4120351-f702-437b-aa9b-40979fa30455-kube-api-access-cbsll\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.209314 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.209279 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.276530 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.276473 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" podStartSLOduration=2.276458735 podStartE2EDuration="2.276458735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:27.274642241 +0000 UTC m=+984.408271271" watchObservedRunningTime="2026-11-03 18:02:27.276458735 +0000 UTC m=+984.410087807" Nov 03 18:02:27.310338 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.310311 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4120351-f702-437b-aa9b-40979fa30455-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.310555 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.310454 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.310555 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.310487 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4120351-f702-437b-aa9b-40979fa30455-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.310555 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.310515 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.310719 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.310570 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4120351-f702-437b-aa9b-40979fa30455-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.310719 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.310631 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.310719 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.310663 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.310868 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.310743 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbsll\" (UniqueName: \"kubernetes.io/projected/e4120351-f702-437b-aa9b-40979fa30455-kube-api-access-cbsll\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.310868 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.310780 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.310964 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.310942 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.311082 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.311059 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.311181 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.311162 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.311247 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.311194 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4120351-f702-437b-aa9b-40979fa30455-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.311289 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.311263 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.313318 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.313298 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.313536 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.313517 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4120351-f702-437b-aa9b-40979fa30455-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.368822 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.368788 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbsll\" (UniqueName: \"kubernetes.io/projected/e4120351-f702-437b-aa9b-40979fa30455-kube-api-access-cbsll\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.368822 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.368807 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4120351-f702-437b-aa9b-40979fa30455-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.584006 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.583902 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:27.664848 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.664816 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:27.925058 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:27.924982 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:02:28.098480 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:28.098444 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t"] Nov 03 18:02:28.103018 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:02:28.102985 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4120351_f702_437b_aa9b_40979fa30455.slice/crio-b8a6a95fed779b28d2a1a8d34c4708a26a8fd058a6e75b4611d41afcddf56452 WatchSource:0}: Error finding container b8a6a95fed779b28d2a1a8d34c4708a26a8fd058a6e75b4611d41afcddf56452: Status 404 returned error can't find the container with id b8a6a95fed779b28d2a1a8d34c4708a26a8fd058a6e75b4611d41afcddf56452 Nov 03 18:02:28.105412 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:28.105360 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:02:28.105545 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:28.105450 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:02:28.105545 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:28.105494 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:02:28.144621 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:28.144591 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" event={"ID":"e4120351-f702-437b-aa9b-40979fa30455","Type":"ContainerStarted","Data":"b8a6a95fed779b28d2a1a8d34c4708a26a8fd058a6e75b4611d41afcddf56452"} Nov 03 18:02:29.152651 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:29.152606 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" event={"ID":"e4120351-f702-437b-aa9b-40979fa30455","Type":"ContainerStarted","Data":"2cc75fa9b42ceca8837288cdba652af7aadaf3c2919b039b404590272a8bf1f7"} Nov 03 18:02:29.315220 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:29.315171 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" podStartSLOduration=2.315157317 podStartE2EDuration="2.315157317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:29.312459685 +0000 UTC m=+986.446088727" watchObservedRunningTime="2026-11-03 18:02:29.315157317 +0000 UTC m=+986.448786326" Nov 03 18:02:29.665445 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:29.665409 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:29.666833 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:29.666806 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 18:02:29.666975 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:29.666863 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" podUID="e4120351-f702-437b-aa9b-40979fa30455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 18:02:30.665135 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:30.665100 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 18:02:30.665562 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:30.665153 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" podUID="e4120351-f702-437b-aa9b-40979fa30455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 18:02:31.665747 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:31.665710 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 18:02:31.666160 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:31.665763 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" podUID="e4120351-f702-437b-aa9b-40979fa30455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 18:02:32.234659 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:32.234622 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:32.665205 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:32.665170 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 18:02:32.665372 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:32.665219 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" podUID="e4120351-f702-437b-aa9b-40979fa30455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 18:02:32.763950 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:32.763913 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:33.665475 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:33.665440 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 18:02:33.665659 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:33.665496 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" podUID="e4120351-f702-437b-aa9b-40979fa30455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 18:02:34.665820 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:34.665780 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Nov 03 18:02:34.666309 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:34.665844 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" podUID="e4120351-f702-437b-aa9b-40979fa30455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Nov 03 18:02:35.670599 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:35.670571 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:35.671061 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:35.670940 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:35.671627 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:35.671606 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:02:35.924766 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:35.924684 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:02:35.937461 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:35.937436 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:02:36.206315 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:36.206205 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:02:36.836628 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:36.836588 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:38.764273 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:38.764236 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:40.320445 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:40.320359 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:40.525657 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:40.525621 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:40.940302 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:40.940269 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p"] Nov 03 18:02:40.944217 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:40.944194 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:40.988492 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:40.988450 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:40.988668 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:40.988500 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:40.988668 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:40.988650 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:40.988815 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:40.988792 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:40.988892 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:40.988872 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mncjm\" (UniqueName: \"kubernetes.io/projected/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-kube-api-access-mncjm\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:40.988931 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:40.988902 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:40.988975 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:40.988932 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:40.989016 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:40.988983 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:40.989050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:40.989041 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:40.994014 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:40.993986 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-aab753-openshift-default-dockercfg-2fhq6\"" Nov 03 18:02:41.045888 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.045838 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p"] Nov 03 18:02:41.090363 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.090295 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:41.090617 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.090378 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mncjm\" (UniqueName: \"kubernetes.io/projected/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-kube-api-access-mncjm\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:41.090617 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.090450 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:41.090617 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.090493 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:41.090617 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.090522 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:41.090617 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.090583 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:41.090893 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.090651 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:41.090893 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.090680 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:41.090893 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.090759 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:41.090893 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.090788 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:41.091072 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.090896 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:41.091072 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.090992 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:41.091141 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.091101 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:41.091575 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.091552 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:41.093195 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.093165 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:41.093455 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.093439 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:41.148148 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.148106 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:41.148305 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.148158 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mncjm\" (UniqueName: \"kubernetes.io/projected/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-kube-api-access-mncjm\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:41.257015 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.256930 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:41.479847 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.479809 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:41.686788 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.686755 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p"] Nov 03 18:02:41.692937 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:02:41.692911 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d8d1e1_cbb6_4e0f_93ac_9a19035e7681.slice/crio-689f9d68b8118ea201e4588bf0d629465be1d38a02af66354009b8f09eae8324 WatchSource:0}: Error finding container 689f9d68b8118ea201e4588bf0d629465be1d38a02af66354009b8f09eae8324: Status 404 returned error can't find the container with id 689f9d68b8118ea201e4588bf0d629465be1d38a02af66354009b8f09eae8324 Nov 03 18:02:41.695327 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.695298 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:02:41.695442 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.695362 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:02:41.695442 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:41.695389 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:02:42.240424 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:42.239630 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" event={"ID":"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681","Type":"ContainerStarted","Data":"8df3f84577c058838bacc55be4aed36b380468a334031e3bad56e43191e683af"} Nov 03 18:02:42.240424 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:42.239702 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" event={"ID":"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681","Type":"ContainerStarted","Data":"689f9d68b8118ea201e4588bf0d629465be1d38a02af66354009b8f09eae8324"} Nov 03 18:02:42.257474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:42.257435 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:42.259479 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:42.259448 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 18:02:42.259640 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:42.259509 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" podUID="86d8d1e1-cbb6-4e0f-93ac-9a19035e7681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 18:02:42.358976 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:42.358925 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" podStartSLOduration=2.358910878 podStartE2EDuration="2.358910878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:42.357605594 +0000 UTC m=+999.491234632" watchObservedRunningTime="2026-11-03 18:02:42.358910878 +0000 UTC m=+999.492539887" Nov 03 18:02:43.257511 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:43.257477 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 18:02:43.257923 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:43.257526 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" podUID="86d8d1e1-cbb6-4e0f-93ac-9a19035e7681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 18:02:44.257633 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:44.257599 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 18:02:44.258081 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:44.257647 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" podUID="86d8d1e1-cbb6-4e0f-93ac-9a19035e7681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 18:02:45.257192 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:45.257155 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 18:02:45.257369 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:45.257205 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" podUID="86d8d1e1-cbb6-4e0f-93ac-9a19035e7681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 18:02:46.194122 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:46.194079 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:46.258132 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:46.258093 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 18:02:46.258327 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:46.258150 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" podUID="86d8d1e1-cbb6-4e0f-93ac-9a19035e7681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 18:02:47.257892 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:47.257859 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 18:02:47.258305 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:47.257915 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" podUID="86d8d1e1-cbb6-4e0f-93ac-9a19035e7681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 18:02:48.257836 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:48.257792 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 18:02:48.258009 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:48.257879 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" podUID="86d8d1e1-cbb6-4e0f-93ac-9a19035e7681" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 18:02:49.261358 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:49.261329 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:49.261797 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:49.261640 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:49.262459 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:49.262440 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:02:50.870049 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:50.870011 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:55.475768 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:55.475721 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:56.237530 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:02:56.237492 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:02:57.723336 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:57.723302 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:58.573572 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:58.573536 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:02:58.777240 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:02:58.777201 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:03:00.085746 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:00.085714 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:03:04.693325 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:04.693292 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:03:09.192413 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:09.192357 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:03:10.837535 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:10.837496 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:03:12.711936 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:12.711893 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:03:13.873682 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:13.873634 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:03:18.695950 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:18.695911 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:03:19.396281 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:19.396246 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:03:22.357995 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:22.357961 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:03:23.174463 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:23.174425 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:03:26.236284 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:03:26.236249 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:03:27.673689 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:27.673648 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:03:32.292194 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:32.292143 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:03:36.771545 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:36.771508 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:03:41.271718 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:41.271681 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:03:42.854207 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:42.854173 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn"] Nov 03 18:03:42.858561 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:42.858538 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:03:42.953500 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:42.953473 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a86228\"/\"distinct-2-w4-a86228-kserve-self-signed-certs\"" Nov 03 18:03:42.959125 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:42.959089 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn"] Nov 03 18:03:42.999730 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:42.999685 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:03:43.013851 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:43.013812 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-model-cache\") pod \"distinct-2-w4-a86228-kserve-5988444c97-9nqsn\" (UID: \"277a1fec-5c82-449d-839f-e843fa0f813d\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:03:43.014133 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:43.014107 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/277a1fec-5c82-449d-839f-e843fa0f813d-tls-certs\") pod \"distinct-2-w4-a86228-kserve-5988444c97-9nqsn\" (UID: \"277a1fec-5c82-449d-839f-e843fa0f813d\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:03:43.014296 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:43.014260 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvtdp\" (UniqueName: \"kubernetes.io/projected/277a1fec-5c82-449d-839f-e843fa0f813d-kube-api-access-zvtdp\") pod \"distinct-2-w4-a86228-kserve-5988444c97-9nqsn\" (UID: \"277a1fec-5c82-449d-839f-e843fa0f813d\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:03:43.014383 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:43.014321 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-home\") pod \"distinct-2-w4-a86228-kserve-5988444c97-9nqsn\" (UID: \"277a1fec-5c82-449d-839f-e843fa0f813d\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:03:43.014598 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:43.014574 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-dshm\") pod \"distinct-2-w4-a86228-kserve-5988444c97-9nqsn\" (UID: \"277a1fec-5c82-449d-839f-e843fa0f813d\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:03:43.014732 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:43.014714 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-tmp-dir\") pod \"distinct-2-w4-a86228-kserve-5988444c97-9nqsn\" (UID: \"277a1fec-5c82-449d-839f-e843fa0f813d\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:03:43.115751 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:43.115668 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-home\") pod \"distinct-2-w4-a86228-kserve-5988444c97-9nqsn\" (UID: \"277a1fec-5c82-449d-839f-e843fa0f813d\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:03:43.115751 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:43.115733 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-dshm\") pod \"distinct-2-w4-a86228-kserve-5988444c97-9nqsn\" (UID: \"277a1fec-5c82-449d-839f-e843fa0f813d\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:03:43.115961 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:43.115840 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-tmp-dir\") pod \"distinct-2-w4-a86228-kserve-5988444c97-9nqsn\" (UID: \"277a1fec-5c82-449d-839f-e843fa0f813d\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:03:43.115961 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:43.115922 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-model-cache\") pod \"distinct-2-w4-a86228-kserve-5988444c97-9nqsn\" (UID: \"277a1fec-5c82-449d-839f-e843fa0f813d\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:03:43.116070 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:43.115994 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/277a1fec-5c82-449d-839f-e843fa0f813d-tls-certs\") pod \"distinct-2-w4-a86228-kserve-5988444c97-9nqsn\" (UID: \"277a1fec-5c82-449d-839f-e843fa0f813d\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:03:43.116070 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:43.116058 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvtdp\" (UniqueName: \"kubernetes.io/projected/277a1fec-5c82-449d-839f-e843fa0f813d-kube-api-access-zvtdp\") pod \"distinct-2-w4-a86228-kserve-5988444c97-9nqsn\" (UID: \"277a1fec-5c82-449d-839f-e843fa0f813d\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:03:43.116195 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:43.116153 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-home\") pod \"distinct-2-w4-a86228-kserve-5988444c97-9nqsn\" (UID: \"277a1fec-5c82-449d-839f-e843fa0f813d\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:03:43.116305 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:43.116277 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-tmp-dir\") pod \"distinct-2-w4-a86228-kserve-5988444c97-9nqsn\" (UID: \"277a1fec-5c82-449d-839f-e843fa0f813d\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:03:43.116717 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:43.116694 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-model-cache\") pod \"distinct-2-w4-a86228-kserve-5988444c97-9nqsn\" (UID: \"277a1fec-5c82-449d-839f-e843fa0f813d\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:03:43.118424 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:43.118388 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-dshm\") pod \"distinct-2-w4-a86228-kserve-5988444c97-9nqsn\" (UID: \"277a1fec-5c82-449d-839f-e843fa0f813d\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:03:43.118750 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:43.118731 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/277a1fec-5c82-449d-839f-e843fa0f813d-tls-certs\") pod \"distinct-2-w4-a86228-kserve-5988444c97-9nqsn\" (UID: \"277a1fec-5c82-449d-839f-e843fa0f813d\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:03:43.173729 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:43.173695 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvtdp\" (UniqueName: \"kubernetes.io/projected/277a1fec-5c82-449d-839f-e843fa0f813d-kube-api-access-zvtdp\") pod \"distinct-2-w4-a86228-kserve-5988444c97-9nqsn\" (UID: \"277a1fec-5c82-449d-839f-e843fa0f813d\") " pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:03:43.470801 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:43.470718 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:03:43.846089 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:43.846053 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn"] Nov 03 18:03:43.852057 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:03:43.852029 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod277a1fec_5c82_449d_839f_e843fa0f813d.slice/crio-98df22d70fb79b12264bc4e84a3c46e64dd1d42507a03e4575d7cfc4b87ade14 WatchSource:0}: Error finding container 98df22d70fb79b12264bc4e84a3c46e64dd1d42507a03e4575d7cfc4b87ade14: Status 404 returned error can't find the container with id 98df22d70fb79b12264bc4e84a3c46e64dd1d42507a03e4575d7cfc4b87ade14 Nov 03 18:03:44.625252 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:44.625212 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" event={"ID":"277a1fec-5c82-449d-839f-e843fa0f813d","Type":"ContainerStarted","Data":"06cae19bb410a9ea1655bb0e740511db4aba660ce4db15fe1b6889a3d00426fe"} Nov 03 18:03:44.625252 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:44.625255 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" event={"ID":"277a1fec-5c82-449d-839f-e843fa0f813d","Type":"ContainerStarted","Data":"98df22d70fb79b12264bc4e84a3c46e64dd1d42507a03e4575d7cfc4b87ade14"} Nov 03 18:03:44.736523 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:44.736469 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" podStartSLOduration=2.736455593 podStartE2EDuration="2.736455593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:03:44.73466496 +0000 UTC m=+1061.868293998" watchObservedRunningTime="2026-11-03 18:03:44.736455593 +0000 UTC m=+1061.870084602" Nov 03 18:03:45.471108 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:45.471073 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:03:45.776934 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:45.776850 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:03:50.295457 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:50.295422 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:03:53.471304 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:53.471270 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:03:53.483220 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:53.483191 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:03:53.690101 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:53.690069 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:03:54.796963 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:54.796928 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:03:56.235449 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:03:56.235416 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:03:59.262301 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:03:59.262252 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:04:03.768681 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:03.768627 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:04:08.265416 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:08.265366 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:04:11.182844 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:11.182809 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:04:11.384221 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:11.384187 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:04:12.962831 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:12.962743 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:04:17.566384 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:17.566320 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:04:22.058016 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:22.057967 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:04:26.238272 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:04:26.238236 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:04:26.559254 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:26.559225 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:04:27.943999 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:27.943905 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:04:31.064973 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:31.064933 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:04:31.475152 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:31.475053 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:04:35.801187 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:35.801149 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:04:40.381133 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:40.381098 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:04:44.986204 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:44.986170 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:04:45.800353 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:45.800317 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf"] Nov 03 18:04:45.804367 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:45.804346 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:04:45.851833 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:45.851796 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-tmp-dir\") pod \"distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf\" (UID: \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:04:45.852008 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:45.851852 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-model-cache\") pod \"distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf\" (UID: \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:04:45.852084 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:45.852062 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vxws\" (UniqueName: \"kubernetes.io/projected/ec4a7156-46e6-4274-9794-e6d1f14aec3f-kube-api-access-8vxws\") pod \"distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf\" (UID: \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:04:45.852132 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:45.852093 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-dshm\") pod \"distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf\" (UID: \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:04:45.852132 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:45.852122 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec4a7156-46e6-4274-9794-e6d1f14aec3f-tls-certs\") pod \"distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf\" (UID: \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:04:45.852238 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:45.852167 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-home\") pod \"distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf\" (UID: \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:04:45.898991 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:45.898951 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-ef9633\"/\"distinct-2-w1-ef9633-kserve-self-signed-certs\"" Nov 03 18:04:45.901413 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:45.901382 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf"] Nov 03 18:04:45.953900 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:45.953864 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vxws\" (UniqueName: \"kubernetes.io/projected/ec4a7156-46e6-4274-9794-e6d1f14aec3f-kube-api-access-8vxws\") pod \"distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf\" (UID: \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:04:45.954512 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:45.954485 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-dshm\") pod \"distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf\" (UID: \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:04:45.954744 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:45.954725 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec4a7156-46e6-4274-9794-e6d1f14aec3f-tls-certs\") pod \"distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf\" (UID: \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:04:45.954867 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:45.954852 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-home\") pod \"distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf\" (UID: \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:04:45.955019 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:45.954995 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-tmp-dir\") pod \"distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf\" (UID: \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:04:45.955158 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:45.955057 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-model-cache\") pod \"distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf\" (UID: \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:04:45.955578 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:45.955553 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-model-cache\") pod \"distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf\" (UID: \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:04:45.956196 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:45.956173 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-home\") pod \"distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf\" (UID: \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:04:45.956312 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:45.956255 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-tmp-dir\") pod \"distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf\" (UID: \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:04:45.958218 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:45.958198 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-dshm\") pod \"distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf\" (UID: \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:04:45.959368 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:45.959347 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec4a7156-46e6-4274-9794-e6d1f14aec3f-tls-certs\") pod \"distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf\" (UID: \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:04:45.976781 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:45.976747 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:04:46.011963 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:46.011931 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vxws\" (UniqueName: \"kubernetes.io/projected/ec4a7156-46e6-4274-9794-e6d1f14aec3f-kube-api-access-8vxws\") pod \"distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf\" (UID: \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\") " pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:04:46.116436 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:46.116232 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:04:46.548893 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:46.548806 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf"] Nov 03 18:04:46.555597 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:04:46.555572 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec4a7156_46e6_4274_9794_e6d1f14aec3f.slice/crio-c79540f06d5cd02ce00e25e3b12724edd02d1474381b249d720ecf6493be752d WatchSource:0}: Error finding container c79540f06d5cd02ce00e25e3b12724edd02d1474381b249d720ecf6493be752d: Status 404 returned error can't find the container with id c79540f06d5cd02ce00e25e3b12724edd02d1474381b249d720ecf6493be752d Nov 03 18:04:46.992250 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:46.992218 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" event={"ID":"ec4a7156-46e6-4274-9794-e6d1f14aec3f","Type":"ContainerStarted","Data":"d5ab982aff08f0268827f8857a4e6f4598750a2bc60ed39be8c8abe3650e0a5e"} Nov 03 18:04:46.992438 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:46.992256 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" event={"ID":"ec4a7156-46e6-4274-9794-e6d1f14aec3f","Type":"ContainerStarted","Data":"c79540f06d5cd02ce00e25e3b12724edd02d1474381b249d720ecf6493be752d"} Nov 03 18:04:47.103427 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:47.103352 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" podStartSLOduration=2.103339156 podStartE2EDuration="2.103339156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:04:47.100948391 +0000 UTC m=+1124.234577400" watchObservedRunningTime="2026-11-03 18:04:47.103339156 +0000 UTC m=+1124.236968164" Nov 03 18:04:47.116986 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:47.116957 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:04:49.465150 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:49.465112 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:04:51.342515 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:51.342476 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:04:54.083132 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:54.083100 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:04:56.116814 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:56.116780 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:04:56.135983 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:56.135953 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:04:56.235375 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:04:56.235338 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:04:56.833314 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:56.833274 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm"] Nov 03 18:04:56.837254 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:56.837233 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:04:56.902002 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:56.901963 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsf6r\" (UniqueName: \"kubernetes.io/projected/a124942f-4351-4a09-8506-bf2bda9f71a3-kube-api-access-dsf6r\") pod \"distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm\" (UID: \"a124942f-4351-4a09-8506-bf2bda9f71a3\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:04:56.902167 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:56.902127 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-home\") pod \"distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm\" (UID: \"a124942f-4351-4a09-8506-bf2bda9f71a3\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:04:56.902240 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:56.902225 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a124942f-4351-4a09-8506-bf2bda9f71a3-tls-certs\") pod \"distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm\" (UID: \"a124942f-4351-4a09-8506-bf2bda9f71a3\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:04:56.902316 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:56.902292 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-model-cache\") pod \"distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm\" (UID: \"a124942f-4351-4a09-8506-bf2bda9f71a3\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:04:56.902376 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:56.902340 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-dshm\") pod \"distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm\" (UID: \"a124942f-4351-4a09-8506-bf2bda9f71a3\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:04:56.902515 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:56.902439 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-tmp-dir\") pod \"distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm\" (UID: \"a124942f-4351-4a09-8506-bf2bda9f71a3\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:04:56.932328 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:56.932297 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-0adcbc\"/\"distinct-2-w3-0adcbc-kserve-self-signed-certs\"" Nov 03 18:04:56.936691 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:56.936668 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm"] Nov 03 18:04:56.981247 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:56.981212 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:04:57.004046 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:57.004006 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsf6r\" (UniqueName: \"kubernetes.io/projected/a124942f-4351-4a09-8506-bf2bda9f71a3-kube-api-access-dsf6r\") pod \"distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm\" (UID: \"a124942f-4351-4a09-8506-bf2bda9f71a3\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:04:57.004223 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:57.004095 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-home\") pod \"distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm\" (UID: \"a124942f-4351-4a09-8506-bf2bda9f71a3\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:04:57.004223 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:57.004129 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a124942f-4351-4a09-8506-bf2bda9f71a3-tls-certs\") pod \"distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm\" (UID: \"a124942f-4351-4a09-8506-bf2bda9f71a3\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:04:57.004223 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:57.004172 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-model-cache\") pod \"distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm\" (UID: \"a124942f-4351-4a09-8506-bf2bda9f71a3\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:04:57.004223 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:57.004209 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-dshm\") pod \"distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm\" (UID: \"a124942f-4351-4a09-8506-bf2bda9f71a3\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:04:57.004369 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:57.004229 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-tmp-dir\") pod \"distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm\" (UID: \"a124942f-4351-4a09-8506-bf2bda9f71a3\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:04:57.004706 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:57.004686 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-tmp-dir\") pod \"distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm\" (UID: \"a124942f-4351-4a09-8506-bf2bda9f71a3\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:04:57.004751 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:57.004696 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-model-cache\") pod \"distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm\" (UID: \"a124942f-4351-4a09-8506-bf2bda9f71a3\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:04:57.004787 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:57.004749 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-home\") pod \"distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm\" (UID: \"a124942f-4351-4a09-8506-bf2bda9f71a3\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:04:57.006750 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:57.006727 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-dshm\") pod \"distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm\" (UID: \"a124942f-4351-4a09-8506-bf2bda9f71a3\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:04:57.007043 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:57.007027 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a124942f-4351-4a09-8506-bf2bda9f71a3-tls-certs\") pod \"distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm\" (UID: \"a124942f-4351-4a09-8506-bf2bda9f71a3\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:04:57.061264 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:57.061237 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:04:57.061766 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:57.061744 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsf6r\" (UniqueName: \"kubernetes.io/projected/a124942f-4351-4a09-8506-bf2bda9f71a3-kube-api-access-dsf6r\") pod \"distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm\" (UID: \"a124942f-4351-4a09-8506-bf2bda9f71a3\") " pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:04:57.151437 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:57.151337 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:04:57.575075 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:57.575041 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm"] Nov 03 18:04:57.581276 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:04:57.581244 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda124942f_4351_4a09_8506_bf2bda9f71a3.slice/crio-9ea5d208830ad4f1fa6449a87646145d57825f36b9a36d8c6eb03b276a2cd0bb WatchSource:0}: Error finding container 9ea5d208830ad4f1fa6449a87646145d57825f36b9a36d8c6eb03b276a2cd0bb: Status 404 returned error can't find the container with id 9ea5d208830ad4f1fa6449a87646145d57825f36b9a36d8c6eb03b276a2cd0bb Nov 03 18:04:58.059015 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:58.058978 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" event={"ID":"a124942f-4351-4a09-8506-bf2bda9f71a3","Type":"ContainerStarted","Data":"eb331f18634d0b57bf88d0a5527ae4e047220de90cc067ba5e050bfcf0537112"} Nov 03 18:04:58.059199 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:58.059027 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" event={"ID":"a124942f-4351-4a09-8506-bf2bda9f71a3","Type":"ContainerStarted","Data":"9ea5d208830ad4f1fa6449a87646145d57825f36b9a36d8c6eb03b276a2cd0bb"} Nov 03 18:04:58.151486 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:58.151455 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:04:58.171376 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:58.171321 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" podStartSLOduration=2.171308082 podStartE2EDuration="2.171308082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:04:58.169217349 +0000 UTC m=+1135.302846357" watchObservedRunningTime="2026-11-03 18:04:58.171308082 +0000 UTC m=+1135.304937091" Nov 03 18:04:58.567618 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:04:58.567585 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:05:03.087313 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:03.087273 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:05:07.152458 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:07.152418 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:05:07.164922 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:07.164896 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:05:07.611508 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:07.611470 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:05:08.135069 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:08.135038 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:05:11.562299 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:11.562257 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:05:12.248942 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.248902 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:05:12.386538 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.386502 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw"] Nov 03 18:05:12.392341 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.392315 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.447428 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.445735 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-0ccbb9-openshift-default-dockercfg-8dgdl\"" Nov 03 18:05:12.508475 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.506438 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.508475 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.506505 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.508475 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.506581 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.508475 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.506612 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.508475 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.506664 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.508475 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.506710 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.508475 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.506797 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.508475 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.506892 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5030e39-1c83-41e5-9484-f0b9c183084d-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.508475 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.506949 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l66j5\" (UniqueName: \"kubernetes.io/projected/d5030e39-1c83-41e5-9484-f0b9c183084d-kube-api-access-l66j5\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.511672 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.511540 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw"] Nov 03 18:05:12.608828 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.608796 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.609293 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.608866 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5030e39-1c83-41e5-9484-f0b9c183084d-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.609293 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.608919 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l66j5\" (UniqueName: \"kubernetes.io/projected/d5030e39-1c83-41e5-9484-f0b9c183084d-kube-api-access-l66j5\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.609293 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.609013 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.609293 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.609037 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.609293 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.609105 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.609293 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.609133 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.609293 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.609171 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.609293 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.609213 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.609808 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.609784 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.609900 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.609876 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.609977 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.609961 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.610165 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.610147 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.610879 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.610854 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5030e39-1c83-41e5-9484-f0b9c183084d-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.612419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.612373 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.613474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.613447 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.671816 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.671746 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:12.712638 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:12.712593 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l66j5\" (UniqueName: \"kubernetes.io/projected/d5030e39-1c83-41e5-9484-f0b9c183084d-kube-api-access-l66j5\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:13.011379 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:13.011343 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:13.765727 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:13.765695 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw"] Nov 03 18:05:13.771246 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:05:13.771220 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5030e39_1c83_41e5_9484_f0b9c183084d.slice/crio-0035a44802ae7a4173ee6f963ce9de09b3d350c2259ddf5b804bddf5824f9b8e WatchSource:0}: Error finding container 0035a44802ae7a4173ee6f963ce9de09b3d350c2259ddf5b804bddf5824f9b8e: Status 404 returned error can't find the container with id 0035a44802ae7a4173ee6f963ce9de09b3d350c2259ddf5b804bddf5824f9b8e Nov 03 18:05:13.773754 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:13.773720 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:05:13.773838 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:13.773789 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:05:13.773838 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:13.773815 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:05:14.169974 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:14.169933 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" event={"ID":"d5030e39-1c83-41e5-9484-f0b9c183084d","Type":"ContainerStarted","Data":"9f205abcc96a42eec8883c6dbb6cc8b779fa6d702a2750adf4bee81d89fd0c71"} Nov 03 18:05:14.170582 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:14.169988 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" event={"ID":"d5030e39-1c83-41e5-9484-f0b9c183084d","Type":"ContainerStarted","Data":"0035a44802ae7a4173ee6f963ce9de09b3d350c2259ddf5b804bddf5824f9b8e"} Nov 03 18:05:14.324418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:14.321109 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" podStartSLOduration=2.321090352 podStartE2EDuration="2.321090352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:14.320313578 +0000 UTC m=+1151.453942579" watchObservedRunningTime="2026-11-03 18:05:14.321090352 +0000 UTC m=+1151.454719363" Nov 03 18:05:15.012789 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:15.012752 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:15.014364 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:15.014339 5805 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 18:05:15.014492 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:15.014387 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" podUID="d5030e39-1c83-41e5-9484-f0b9c183084d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 18:05:16.012727 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:16.012689 5805 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 18:05:16.012936 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:16.012756 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" podUID="d5030e39-1c83-41e5-9484-f0b9c183084d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 18:05:17.009279 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:17.009237 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:05:17.012372 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:17.012347 5805 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 18:05:17.012610 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:17.012582 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" podUID="d5030e39-1c83-41e5-9484-f0b9c183084d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 18:05:18.012834 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:18.012794 5805 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 18:05:18.013235 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:18.012850 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" podUID="d5030e39-1c83-41e5-9484-f0b9c183084d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 18:05:19.012406 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:19.012371 5805 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 18:05:19.012591 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:19.012447 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" podUID="d5030e39-1c83-41e5-9484-f0b9c183084d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 18:05:20.012557 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:20.012517 5805 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 03 18:05:20.012963 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:20.012596 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" podUID="d5030e39-1c83-41e5-9484-f0b9c183084d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 03 18:05:21.017007 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:21.016977 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:21.017487 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:21.017282 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:21.018358 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:21.018335 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:05:21.702649 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:21.702613 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:05:26.236212 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:05:26.236168 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:05:26.381909 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.381874 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:05:26.533526 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.533444 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz"] Nov 03 18:05:26.538776 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.538754 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.634285 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.634249 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz"] Nov 03 18:05:26.635337 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.634925 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz8bb\" (UniqueName: \"kubernetes.io/projected/984bc848-d32b-43d0-92cb-702cb3fa8f87-kube-api-access-fz8bb\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.635337 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.634990 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.635337 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.635092 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.635337 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.635115 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.635337 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.635142 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.635337 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.635339 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.635726 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.635438 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984bc848-d32b-43d0-92cb-702cb3fa8f87-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.635726 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.635469 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.635726 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.635518 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.736604 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.736572 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.736790 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.736613 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz8bb\" (UniqueName: \"kubernetes.io/projected/984bc848-d32b-43d0-92cb-702cb3fa8f87-kube-api-access-fz8bb\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.736790 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.736645 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.736790 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.736717 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.736790 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.736740 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.736790 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.736766 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.737029 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.736834 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.737029 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.736893 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984bc848-d32b-43d0-92cb-702cb3fa8f87-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.737029 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.736921 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.737240 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.737215 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.737472 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.737355 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.737472 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.737452 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.737597 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.737576 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.737719 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.737696 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984bc848-d32b-43d0-92cb-702cb3fa8f87-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.739001 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.738981 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.739084 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.739030 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.795326 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.795260 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.795326 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.795315 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz8bb\" (UniqueName: \"kubernetes.io/projected/984bc848-d32b-43d0-92cb-702cb3fa8f87-kube-api-access-fz8bb\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.852391 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.852354 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:26.866012 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.865985 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw"] Nov 03 18:05:26.870202 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.870183 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:26.937549 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.937501 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw"] Nov 03 18:05:26.938549 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.938523 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:26.938831 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.938815 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:26.939038 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.939020 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:26.939202 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.939186 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:26.939442 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.939382 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:26.939579 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.939564 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22nvw\" (UniqueName: \"kubernetes.io/projected/35c5ded9-15bd-4f4f-b09e-0e91231efd07-kube-api-access-22nvw\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:26.939742 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.939726 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:26.939919 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.939903 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:26.940096 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:26.940081 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:27.041137 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.041102 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:27.041286 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.041175 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:27.041286 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.041211 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:27.041286 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.041279 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:27.041487 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.041321 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22nvw\" (UniqueName: \"kubernetes.io/projected/35c5ded9-15bd-4f4f-b09e-0e91231efd07-kube-api-access-22nvw\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:27.041487 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.041359 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:27.041487 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.041442 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:27.041487 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.041475 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:27.041694 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.041574 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:27.041694 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.041594 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:27.041781 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.041690 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:27.041781 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.041752 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:27.042271 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.042240 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:27.042806 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.042783 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:27.044699 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.044674 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:27.045013 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.044989 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:27.098841 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.098812 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:27.100495 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.100474 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22nvw\" (UniqueName: \"kubernetes.io/projected/35c5ded9-15bd-4f4f-b09e-0e91231efd07-kube-api-access-22nvw\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:27.189973 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.189944 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:27.284970 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.284927 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz"] Nov 03 18:05:27.291360 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:05:27.291327 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984bc848_d32b_43d0_92cb_702cb3fa8f87.slice/crio-504ec44e7cebd0aebf0237b4fc97a038a2af21e3b946fb998a394614b6de97cd WatchSource:0}: Error finding container 504ec44e7cebd0aebf0237b4fc97a038a2af21e3b946fb998a394614b6de97cd: Status 404 returned error can't find the container with id 504ec44e7cebd0aebf0237b4fc97a038a2af21e3b946fb998a394614b6de97cd Nov 03 18:05:27.294242 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.294211 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:05:27.294365 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.294282 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:05:27.294365 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.294310 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:05:27.574077 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.574041 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw"] Nov 03 18:05:27.579336 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:05:27.579307 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c5ded9_15bd_4f4f_b09e_0e91231efd07.slice/crio-04598f730b1a3ec2e952715aa99cdaf431274686814a184d5a6418829b8b90b5 WatchSource:0}: Error finding container 04598f730b1a3ec2e952715aa99cdaf431274686814a184d5a6418829b8b90b5: Status 404 returned error can't find the container with id 04598f730b1a3ec2e952715aa99cdaf431274686814a184d5a6418829b8b90b5 Nov 03 18:05:27.582142 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.582106 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:05:27.582252 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.582171 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:05:27.582252 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:27.582219 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:05:28.280304 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:28.280202 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" event={"ID":"984bc848-d32b-43d0-92cb-702cb3fa8f87","Type":"ContainerStarted","Data":"9830bd97d1bf5cd80ed4a807152b8bf20ead7df7afe8deb2e8034f1d0cf08301"} Nov 03 18:05:28.280304 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:28.280269 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" event={"ID":"984bc848-d32b-43d0-92cb-702cb3fa8f87","Type":"ContainerStarted","Data":"504ec44e7cebd0aebf0237b4fc97a038a2af21e3b946fb998a394614b6de97cd"} Nov 03 18:05:28.281923 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:28.281900 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" event={"ID":"35c5ded9-15bd-4f4f-b09e-0e91231efd07","Type":"ContainerStarted","Data":"77dbbde3602bfd27bb3cec51e53d7d4559bdf9d3e6bbeea8d18c43f18a7d5671"} Nov 03 18:05:28.282119 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:28.281941 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" event={"ID":"35c5ded9-15bd-4f4f-b09e-0e91231efd07","Type":"ContainerStarted","Data":"04598f730b1a3ec2e952715aa99cdaf431274686814a184d5a6418829b8b90b5"} Nov 03 18:05:28.406170 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:28.406109 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" podStartSLOduration=2.406094493 podStartE2EDuration="2.406094493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:28.403446972 +0000 UTC m=+1165.537075995" watchObservedRunningTime="2026-11-03 18:05:28.406094493 +0000 UTC m=+1165.539723503" Nov 03 18:05:28.565482 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:28.543859 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" podStartSLOduration=2.543840298 podStartE2EDuration="2.543840298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:28.542871436 +0000 UTC m=+1165.676500477" watchObservedRunningTime="2026-11-03 18:05:28.543840298 +0000 UTC m=+1165.677469310" Nov 03 18:05:28.867537 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:28.862766 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:28.867829 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:28.867802 5805 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 18:05:28.867994 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:28.867969 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" podUID="984bc848-d32b-43d0-92cb-702cb3fa8f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 18:05:29.191229 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:29.191151 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:29.192292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:29.192273 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 18:05:29.192346 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:29.192314 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" podUID="35c5ded9-15bd-4f4f-b09e-0e91231efd07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 18:05:29.853417 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:29.852816 5805 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 18:05:29.853417 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:29.852879 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" podUID="984bc848-d32b-43d0-92cb-702cb3fa8f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 18:05:30.190730 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:30.190636 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 18:05:30.190730 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:30.190690 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" podUID="35c5ded9-15bd-4f4f-b09e-0e91231efd07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 18:05:30.864432 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:30.861469 5805 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 18:05:30.864432 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:30.861529 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" podUID="984bc848-d32b-43d0-92cb-702cb3fa8f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 18:05:30.970417 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:30.970009 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:05:31.190719 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:31.190637 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 18:05:31.190934 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:31.190910 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" podUID="35c5ded9-15bd-4f4f-b09e-0e91231efd07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 18:05:31.680318 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:31.680267 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:05:31.853418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:31.852802 5805 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 18:05:31.853418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:31.852867 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" podUID="984bc848-d32b-43d0-92cb-702cb3fa8f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 18:05:31.880485 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:31.880449 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:05:32.190736 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:32.190695 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 18:05:32.190916 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:32.190754 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" podUID="35c5ded9-15bd-4f4f-b09e-0e91231efd07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 18:05:32.856420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:32.854821 5805 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 18:05:32.856420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:32.854884 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" podUID="984bc848-d32b-43d0-92cb-702cb3fa8f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 18:05:33.190537 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:33.190451 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 18:05:33.190537 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:33.190520 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" podUID="35c5ded9-15bd-4f4f-b09e-0e91231efd07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 18:05:33.853470 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:33.853427 5805 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 03 18:05:33.853648 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:33.853502 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" podUID="984bc848-d32b-43d0-92cb-702cb3fa8f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 03 18:05:34.191173 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:34.191087 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 03 18:05:34.191173 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:34.191149 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" podUID="35c5ded9-15bd-4f4f-b09e-0e91231efd07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 03 18:05:34.857805 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:34.857765 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:34.858113 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:34.858084 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:34.858964 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:34.858945 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:05:35.196348 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:35.196246 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:35.196881 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:35.196633 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:35.198037 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:35.198015 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:05:35.563699 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:35.563667 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:05:40.077438 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:40.077375 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:05:44.557704 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:44.557597 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:05:49.192679 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:49.190527 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:05:53.681556 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:53.681515 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:05:56.237343 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:05:56.237309 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:05:58.201384 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:05:58.201350 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:02.676042 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:02.676007 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:03.168794 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:03.168763 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 18:06:03.170953 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:03.170931 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 18:06:07.181274 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:07.181233 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:11.932105 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:11.932071 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:12.143165 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.143131 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:12.222148 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.222069 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8"] Nov 03 18:06:12.226344 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.226310 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.320747 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.320699 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.320941 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.320756 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.320941 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.320922 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.321066 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.321014 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.321066 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.321047 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.321234 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.321205 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.321382 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.321246 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b450130-5eea-4fe2-896c-62ba2fe84807-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.321382 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.321301 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n4s8\" (UniqueName: \"kubernetes.io/projected/0b450130-5eea-4fe2-896c-62ba2fe84807-kube-api-access-9n4s8\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.321522 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.321386 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.329423 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.325293 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8"] Nov 03 18:06:12.423043 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.423007 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.423211 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.423060 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.423211 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.423123 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.423211 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.423159 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.423371 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.423241 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.423371 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.423265 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b450130-5eea-4fe2-896c-62ba2fe84807-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.423371 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.423315 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n4s8\" (UniqueName: \"kubernetes.io/projected/0b450130-5eea-4fe2-896c-62ba2fe84807-kube-api-access-9n4s8\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.423553 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.423383 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.423553 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.423527 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.423705 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.423679 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-workload-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.423761 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.423711 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-credential-socket\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.423816 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.423778 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-data\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.423930 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.423906 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-workload-certs\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.424068 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.424052 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b450130-5eea-4fe2-896c-62ba2fe84807-istiod-ca-cert\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.425747 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.425719 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-envoy\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.425747 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.425742 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-podinfo\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.480828 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.480745 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-token\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.480828 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.480824 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n4s8\" (UniqueName: \"kubernetes.io/projected/0b450130-5eea-4fe2-896c-62ba2fe84807-kube-api-access-9n4s8\") pod \"e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.531613 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.531575 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96"] Nov 03 18:06:12.536938 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.536909 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.541138 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.541117 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:12.553021 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.552992 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77"] Nov 03 18:06:12.568697 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.568656 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.574281 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.572135 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f"] Nov 03 18:06:12.577921 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.577899 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.594603 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.594576 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96"] Nov 03 18:06:12.644324 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.644278 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f"] Nov 03 18:06:12.661801 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.660976 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77"] Nov 03 18:06:12.726570 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.726534 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlsdc\" (UniqueName: \"kubernetes.io/projected/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-kube-api-access-tlsdc\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.726570 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.726572 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zwhx\" (UniqueName: \"kubernetes.io/projected/aca6224c-f93a-4d51-b37e-f3368efa25dd-kube-api-access-5zwhx\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.726784 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.726595 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.726784 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.726711 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.726784 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.726747 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.726784 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.726764 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.726928 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.726845 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.726928 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.726893 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.726998 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.726929 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca6224c-f93a-4d51-b37e-f3368efa25dd-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.726998 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.726947 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.727072 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.727013 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.727072 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.727051 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.727140 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.727075 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.727140 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.727091 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.727140 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.727127 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.727295 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.727204 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.727295 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.727224 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.727295 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.727246 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.727295 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.727277 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.727461 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.727346 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.727461 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.727423 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.727524 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.727464 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.727524 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.727490 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.727608 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.727548 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.727608 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.727568 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.727695 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.727628 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72c24\" (UniqueName: \"kubernetes.io/projected/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-kube-api-access-72c24\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.727695 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.727651 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.828412 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.828364 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlsdc\" (UniqueName: \"kubernetes.io/projected/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-kube-api-access-tlsdc\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.828600 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.828420 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zwhx\" (UniqueName: \"kubernetes.io/projected/aca6224c-f93a-4d51-b37e-f3368efa25dd-kube-api-access-5zwhx\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.828600 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.828444 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.828600 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.828469 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.828600 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.828495 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.828835 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.828594 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.828835 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.828661 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.828835 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.828696 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.828835 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.828758 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca6224c-f93a-4d51-b37e-f3368efa25dd-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.828835 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.828788 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.828835 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.828829 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.829138 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.828889 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.829138 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.828923 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.829138 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.828936 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.829138 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.828974 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.829138 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.829027 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.829138 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.829086 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.829138 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.829115 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.829562 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.829140 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.829562 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.829190 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.829562 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.829233 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.829562 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.829275 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.829562 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.829337 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.829562 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.829358 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.829562 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.829365 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.829562 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.829451 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.829562 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.829497 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.829562 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.829508 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.829562 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.829526 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.830127 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.829695 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.830127 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.829732 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.830127 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.829734 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca6224c-f93a-4d51-b37e-f3368efa25dd-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.830127 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.829776 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-workload-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.830127 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.829812 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72c24\" (UniqueName: \"kubernetes.io/projected/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-kube-api-access-72c24\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.830127 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.829995 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.830491 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.830228 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.830491 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.830371 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.830491 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.830436 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.830669 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.830566 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-credential-socket\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.830731 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.830697 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istiod-ca-cert\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.830731 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.830709 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-data\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.830842 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.830801 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-workload-certs\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.831319 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.831291 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.831462 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.831388 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.831977 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.831953 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.832162 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.832141 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-envoy\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.832616 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.832598 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.833168 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.833153 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-podinfo\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.886273 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.886234 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlsdc\" (UniqueName: \"kubernetes.io/projected/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-kube-api-access-tlsdc\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.886483 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.886357 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zwhx\" (UniqueName: \"kubernetes.io/projected/aca6224c-f93a-4d51-b37e-f3368efa25dd-kube-api-access-5zwhx\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.886624 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.886601 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.886666 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.886606 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72c24\" (UniqueName: \"kubernetes.io/projected/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-kube-api-access-72c24\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.887592 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.887571 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.888527 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.888511 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:12.890761 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.890739 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-token\") pod \"e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:12.895940 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.895921 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:12.995031 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:12.992800 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8"] Nov 03 18:06:13.003206 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.003173 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:06:13.003311 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.003295 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:06:13.003355 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.003334 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:06:13.153530 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.153439 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:13.286446 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.281015 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77"] Nov 03 18:06:13.298108 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:06:13.298071 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0133a1d6_2676_4f88_8fee_1dfe9c2b28ac.slice/crio-340e5847007adcd59f97c6a84d6588c03de5d1cb8c0c50e40e93f687efbddc49 WatchSource:0}: Error finding container 340e5847007adcd59f97c6a84d6588c03de5d1cb8c0c50e40e93f687efbddc49: Status 404 returned error can't find the container with id 340e5847007adcd59f97c6a84d6588c03de5d1cb8c0c50e40e93f687efbddc49 Nov 03 18:06:13.304716 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.304680 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:06:13.304823 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.304766 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:06:13.304901 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.304809 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:06:13.329594 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.329565 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f"] Nov 03 18:06:13.340814 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.340777 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:06:13.340902 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.340869 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:06:13.340942 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.340914 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:06:13.536927 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.536856 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96"] Nov 03 18:06:13.543274 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:06:13.543240 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df51a82_bd64_4ad8_a164_d5af9ee53e2d.slice/crio-6bdaee003e362c17415496afeaaa9df90313ac27b7534c1da566e93756d28b05 WatchSource:0}: Error finding container 6bdaee003e362c17415496afeaaa9df90313ac27b7534c1da566e93756d28b05: Status 404 returned error can't find the container with id 6bdaee003e362c17415496afeaaa9df90313ac27b7534c1da566e93756d28b05 Nov 03 18:06:13.546507 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.546476 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:06:13.546591 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.546540 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:06:13.546591 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.546574 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:06:13.588986 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.588947 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" event={"ID":"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac","Type":"ContainerStarted","Data":"06e131755902e8c425619882fb0456acde935b06267c090ead3ca977abd0d624"} Nov 03 18:06:13.589151 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.589017 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" event={"ID":"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac","Type":"ContainerStarted","Data":"340e5847007adcd59f97c6a84d6588c03de5d1cb8c0c50e40e93f687efbddc49"} Nov 03 18:06:13.592605 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.592500 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" event={"ID":"aca6224c-f93a-4d51-b37e-f3368efa25dd","Type":"ContainerStarted","Data":"8e5cf7007df00719ab605d265227147d1397bb6ce550f9ce96c3492d4bcb91c4"} Nov 03 18:06:13.592605 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.592562 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" event={"ID":"aca6224c-f93a-4d51-b37e-f3368efa25dd","Type":"ContainerStarted","Data":"8c168f0495eac16288c9faa0960aa0a95ba824941afbd985722167d9c50afac5"} Nov 03 18:06:13.594389 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.594306 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" event={"ID":"2df51a82-bd64-4ad8-a164-d5af9ee53e2d","Type":"ContainerStarted","Data":"6bdaee003e362c17415496afeaaa9df90313ac27b7534c1da566e93756d28b05"} Nov 03 18:06:13.596027 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.596003 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" event={"ID":"0b450130-5eea-4fe2-896c-62ba2fe84807","Type":"ContainerStarted","Data":"8d424cce82c533303c9f10ff5040190a28b0ee508202d0bba49d20527c0b4094"} Nov 03 18:06:13.596149 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.596043 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" event={"ID":"0b450130-5eea-4fe2-896c-62ba2fe84807","Type":"ContainerStarted","Data":"130f667332f9c8cad93e8910a70705d2708c0af50dc5cef01111ab08c4415570"} Nov 03 18:06:13.725424 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.720889 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" podStartSLOduration=1.720871406 podStartE2EDuration="1.720871406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:13.715975675 +0000 UTC m=+1210.849604686" watchObservedRunningTime="2026-11-03 18:06:13.720871406 +0000 UTC m=+1210.854500417" Nov 03 18:06:13.776045 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.776012 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26"] Nov 03 18:06:13.781002 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.780978 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.808435 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.808382 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk"] Nov 03 18:06:13.812725 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.812704 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.831793 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.831741 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" podStartSLOduration=1.83172175 podStartE2EDuration="1.83172175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:13.831094841 +0000 UTC m=+1210.964723877" watchObservedRunningTime="2026-11-03 18:06:13.83172175 +0000 UTC m=+1210.965350763" Nov 03 18:06:13.868419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.856465 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.868744 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.868713 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.868953 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.868932 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.869073 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.869056 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xvhz\" (UniqueName: \"kubernetes.io/projected/e1392256-ac5e-4f9a-90da-87b002ebd848-kube-api-access-9xvhz\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.869272 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.869253 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.869490 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.869465 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.869587 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.869505 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.869587 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.869574 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.869693 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.869625 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.869693 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.869678 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.869796 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.869716 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.869796 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.869750 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1392256-ac5e-4f9a-90da-87b002ebd848-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.869896 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.869793 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.869896 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.869831 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.869997 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.869907 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.869997 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.869942 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nvqj\" (UniqueName: \"kubernetes.io/projected/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-kube-api-access-4nvqj\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.870101 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.870004 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.870101 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.870036 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.889212 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.889173 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:13.893444 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.890827 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26"] Nov 03 18:06:13.893573 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.893558 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk"] Nov 03 18:06:13.896432 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.896413 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:13.951527 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.951471 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" podStartSLOduration=1.951456818 podStartE2EDuration="1.951456818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:13.948637006 +0000 UTC m=+1211.082266046" watchObservedRunningTime="2026-11-03 18:06:13.951456818 +0000 UTC m=+1211.085085827" Nov 03 18:06:13.971336 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.971309 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xvhz\" (UniqueName: \"kubernetes.io/projected/e1392256-ac5e-4f9a-90da-87b002ebd848-kube-api-access-9xvhz\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.971537 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.971370 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.971537 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.971443 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.971537 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.971462 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.971537 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.971505 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.971537 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.971522 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.971787 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.971555 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.971787 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.971588 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.971787 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.971621 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1392256-ac5e-4f9a-90da-87b002ebd848-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.971787 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.971664 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.971787 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.971694 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.971787 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.971757 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.972067 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.971788 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nvqj\" (UniqueName: \"kubernetes.io/projected/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-kube-api-access-4nvqj\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.972067 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.971818 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.972067 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.971845 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.972067 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.971915 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.972067 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.971919 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.972067 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.971955 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.972067 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.972014 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.972067 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.972040 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.972411 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.971955 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.972468 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.972422 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1392256-ac5e-4f9a-90da-87b002ebd848-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.972468 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.972427 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.972776 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.972751 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-data\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.972904 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.972802 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-workload-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.972904 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.972827 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-credential-socket\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.973165 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.973140 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-workload-certs\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.973530 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.973506 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istiod-ca-cert\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.974605 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.974582 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.975173 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.975151 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-envoy\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:13.975266 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.975218 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:13.975450 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:13.975430 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-podinfo\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:14.028965 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:14.028926 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:14.029516 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:14.028940 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-token\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:14.029516 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:14.029047 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xvhz\" (UniqueName: \"kubernetes.io/projected/e1392256-ac5e-4f9a-90da-87b002ebd848-kube-api-access-9xvhz\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:14.030385 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:14.030357 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nvqj\" (UniqueName: \"kubernetes.io/projected/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-kube-api-access-4nvqj\") pod \"e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:14.101683 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:14.101640 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:14.141417 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:14.133659 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:14.611478 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:14.568198 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:14.611478 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:14.581543 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 18:06:14.611478 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:14.581624 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" podUID="0b450130-5eea-4fe2-896c-62ba2fe84807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 18:06:14.734956 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:14.734904 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" event={"ID":"2df51a82-bd64-4ad8-a164-d5af9ee53e2d","Type":"ContainerStarted","Data":"d7a7f658b2d633e3ae1325702d98b3cede880463b91edc9d73a345404110fb20"} Nov 03 18:06:14.929471 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:14.929167 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 18:06:14.929471 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:14.929242 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" podUID="aca6224c-f93a-4d51-b37e-f3368efa25dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 18:06:14.940604 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:14.940549 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 18:06:14.940932 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:14.940887 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" podUID="0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 18:06:15.024419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.006854 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" podStartSLOduration=3.006830572 podStartE2EDuration="3.006830572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:14.904725781 +0000 UTC m=+1212.038354843" watchObservedRunningTime="2026-11-03 18:06:15.006830572 +0000 UTC m=+1212.140459584" Nov 03 18:06:15.185507 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.172327 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 18:06:15.185507 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.172408 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" podUID="2df51a82-bd64-4ad8-a164-d5af9ee53e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 18:06:15.226855 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.204998 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:06:15.226855 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.205083 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:06:15.226855 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.205126 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:06:15.323420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.305299 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:15.338538 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.338505 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26"] Nov 03 18:06:15.339012 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.338767 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk"] Nov 03 18:06:15.406680 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.406627 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:06:15.407123 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.407088 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:06:15.407318 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.407297 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:06:15.546275 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.545810 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 18:06:15.546275 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.545890 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" podUID="0b450130-5eea-4fe2-896c-62ba2fe84807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 18:06:15.587910 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.587577 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" podUID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:15.644794 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.644483 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 18:06:15.701619 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.701523 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" podUID="277a1fec-5c82-449d-839f-e843fa0f813d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:15.752492 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.745289 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 03 18:06:15.874417 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.872816 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" event={"ID":"e1392256-ac5e-4f9a-90da-87b002ebd848","Type":"ContainerStarted","Data":"1303615f124d5bcef7c250aabc5779b027b9ba5a47782f13ccfcf2093442d74c"} Nov 03 18:06:15.875476 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.875445 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" event={"ID":"94ef80bb-de9b-4163-9db7-0fd6f19a8eca","Type":"ContainerStarted","Data":"cd7977001742f84dc86b4de84b5581b779cd0a55fe780e59eb5aacd9b0a61f2f"} Nov 03 18:06:15.891172 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.890792 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 18:06:15.891463 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.891431 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" podUID="0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 18:06:15.903490 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.896497 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 18:06:15.903490 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.896540 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" podUID="aca6224c-f93a-4d51-b37e-f3368efa25dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 18:06:15.935375 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:15.935335 5805 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:16.065519 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.065475 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" podUID="ec4a7156-46e6-4274-9794-e6d1f14aec3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:16.068292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.066865 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:16.137462 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.135495 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 18:06:16.155234 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.155120 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 18:06:16.155234 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.155176 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" podUID="2df51a82-bd64-4ad8-a164-d5af9ee53e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 18:06:16.330419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.327473 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fd8d1c6-xgqjv"] Nov 03 18:06:16.374655 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.374620 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fd8d1c6-xgqjv"] Nov 03 18:06:16.374937 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.374924 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd8d1c6-xgqjv" Nov 03 18:06:16.475980 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.473420 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\"" Nov 03 18:06:16.475980 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.473509 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:06:16.504056 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.480264 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 03 18:06:16.504589 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.497310 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fd8d1c6-xgqjv\" (UID: \"3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd8d1c6-xgqjv" Nov 03 18:06:16.505079 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.505058 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c28hc\" (UniqueName: \"kubernetes.io/projected/3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8-kube-api-access-c28hc\") pod \"maas-api-revoke-sub-9fd8d1c6-xgqjv\" (UID: \"3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd8d1c6-xgqjv" Nov 03 18:06:16.505203 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.505189 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fd8d1c6-xgqjv\" (UID: \"3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd8d1c6-xgqjv" Nov 03 18:06:16.568237 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.568202 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 18:06:16.568513 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.568480 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" podUID="0b450130-5eea-4fe2-896c-62ba2fe84807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 18:06:16.632424 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.607199 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c28hc\" (UniqueName: \"kubernetes.io/projected/3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8-kube-api-access-c28hc\") pod \"maas-api-revoke-sub-9fd8d1c6-xgqjv\" (UID: \"3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd8d1c6-xgqjv" Nov 03 18:06:16.632424 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.621480 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fd8d1c6-xgqjv\" (UID: \"3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd8d1c6-xgqjv" Nov 03 18:06:16.632424 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.621594 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fd8d1c6-xgqjv\" (UID: \"3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd8d1c6-xgqjv" Nov 03 18:06:16.632424 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.622817 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fd8d1c6-xgqjv\" (UID: \"3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd8d1c6-xgqjv" Nov 03 18:06:16.767425 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.672843 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 18:06:16.767425 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.704033 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:06:16.767425 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.763630 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:16.767425 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.764004 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 18:06:16.792419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.787372 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c28hc\" (UniqueName: \"kubernetes.io/projected/3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8-kube-api-access-c28hc\") pod \"maas-api-revoke-sub-9fd8d1c6-xgqjv\" (UID: \"3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd8d1c6-xgqjv" Nov 03 18:06:16.826420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.825894 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fd8d1c6-xgqjv\" (UID: \"3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd8d1c6-xgqjv" Nov 03 18:06:16.934420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.916002 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 18:06:16.934420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.916057 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" podUID="aca6224c-f93a-4d51-b37e-f3368efa25dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 18:06:16.992417 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:16.989741 5805 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" podUID="2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Nov 03 18:06:17.069475 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:17.069420 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:17.084418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:17.069867 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" podUID="ec4a7156-46e6-4274-9794-e6d1f14aec3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:17.123422 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:17.123062 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd8d1c6-xgqjv" Nov 03 18:06:17.299159 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:17.185601 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 18:06:17.299159 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:17.185662 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" podUID="2df51a82-bd64-4ad8-a164-d5af9ee53e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 18:06:17.299159 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:17.229572 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" podUID="e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:17.299159 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:17.270862 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 18:06:17.299159 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:17.270922 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" podUID="0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 18:06:17.346822 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:17.339357 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" event={"ID":"e1392256-ac5e-4f9a-90da-87b002ebd848","Type":"ContainerStarted","Data":"6d3a13a90e4fb354c9350b9ef157e101ede74be0cf2e0fc030532325f2bf02bf"} Nov 03 18:06:17.346822 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:17.339484 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 18:06:17.346822 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:17.339533 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:06:17.381713 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:17.381677 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 18:06:17.489459 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:17.464288 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" podStartSLOduration=4.464267414 podStartE2EDuration="4.464267414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:17.459999327 +0000 UTC m=+1214.593628337" watchObservedRunningTime="2026-11-03 18:06:17.464267414 +0000 UTC m=+1214.597896427" Nov 03 18:06:17.552069 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:17.550243 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 18:06:17.552069 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:17.550298 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" podUID="0b450130-5eea-4fe2-896c-62ba2fe84807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 18:06:17.552069 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:17.550842 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:17.552069 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:17.551083 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 18:06:17.559721 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:17.559125 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" event={"ID":"94ef80bb-de9b-4163-9db7-0fd6f19a8eca","Type":"ContainerStarted","Data":"5ce052d145555942ce3cd15c2961039306daa4df02f4f26c51e2b2f4d2111f55"} Nov 03 18:06:17.568585 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:17.568193 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" podUID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 03 18:06:17.732591 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:17.715683 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 18:06:18.141482 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.138841 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 18:06:18.182420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.178663 5805 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:18.260420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.189762 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:18.260420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.216938 5805 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded" Nov 03 18:06:18.260420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.217208 5805 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" podUID="e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:18.260420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.247056 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:06:18.260420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.247112 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:18.260420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.247160 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:18.260420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.250450 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 18:06:18.260420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.250507 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" podUID="aca6224c-f93a-4d51-b37e-f3368efa25dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 18:06:18.260420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.252612 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" podUID="2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 18:06:18.333429 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.320719 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 18:06:18.333429 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.320798 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" podUID="e1392256-ac5e-4f9a-90da-87b002ebd848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 18:06:18.363057 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.338974 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:18.363057 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.360806 5805 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" podUID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded" Nov 03 18:06:18.404314 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.402704 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 18:06:18.404314 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.402769 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" podUID="2df51a82-bd64-4ad8-a164-d5af9ee53e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 18:06:18.404314 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.403609 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" podUID="e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 03 18:06:18.415448 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.405707 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 18:06:18.439427 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.435240 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 18:06:18.439427 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.435310 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" podUID="0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 18:06:18.465692 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.465124 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 18:06:18.465692 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.465214 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" podUID="94ef80bb-de9b-4163-9db7-0fd6f19a8eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 18:06:18.522209 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.509797 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:18.574818 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.574767 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 03 18:06:18.575835 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.575808 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fd8d1c6-xgqjv"] Nov 03 18:06:18.590837 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.590793 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 18:06:18.591030 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.590857 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" podUID="0b450130-5eea-4fe2-896c-62ba2fe84807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 18:06:18.591030 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:06:18.590954 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e0e7d5a_927e_4de3_af8e_fdc1aacbc3a8.slice/crio-8af86d88ba7c92980e7131329212b0b5a69f3adefe863ac70f9a6e670c77c633 WatchSource:0}: Error finding container 8af86d88ba7c92980e7131329212b0b5a69f3adefe863ac70f9a6e670c77c633: Status 404 returned error can't find the container with id 8af86d88ba7c92980e7131329212b0b5a69f3adefe863ac70f9a6e670c77c633 Nov 03 18:06:18.595415 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.595298 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" podUID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:18.640431 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.619330 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 18:06:18.739378 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.735584 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 18:06:18.811422 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.794517 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" podUID="277a1fec-5c82-449d-839f-e843fa0f813d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 18:06:18.814014 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.813973 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd8d1c6-xgqjv" event={"ID":"3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8","Type":"ContainerStarted","Data":"8af86d88ba7c92980e7131329212b0b5a69f3adefe863ac70f9a6e670c77c633"} Nov 03 18:06:18.904437 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.894451 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 18:06:18.904437 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:18.894514 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" podUID="0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 18:06:19.122620 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.105866 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 18:06:19.122620 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.105921 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" podUID="aca6224c-f93a-4d51-b37e-f3368efa25dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 18:06:19.134837 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.123554 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" podUID="ec4a7156-46e6-4274-9794-e6d1f14aec3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:19.283183 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.283024 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" podStartSLOduration=6.282999678 podStartE2EDuration="6.282999678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:18.710668059 +0000 UTC m=+1215.844297069" watchObservedRunningTime="2026-11-03 18:06:19.282999678 +0000 UTC m=+1216.416628691" Nov 03 18:06:19.333854 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.333814 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 18:06:19.334050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.333879 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" podUID="e1392256-ac5e-4f9a-90da-87b002ebd848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 18:06:19.368433 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.359287 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 18:06:19.381002 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.370556 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 18:06:19.381242 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.381205 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" podUID="94ef80bb-de9b-4163-9db7-0fd6f19a8eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 18:06:19.496422 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.473464 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:19.547420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.520754 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" podUID="e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:19.548291 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.531517 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" podUID="2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 18:06:19.548291 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.546897 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:19.653094 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.652915 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 03 18:06:19.655995 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.655956 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" podUID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:19.684930 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.684894 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 18:06:19.685204 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.685173 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" podUID="0b450130-5eea-4fe2-896c-62ba2fe84807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 18:06:19.685431 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.685413 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 18:06:19.685555 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.685536 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" podUID="2df51a82-bd64-4ad8-a164-d5af9ee53e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 18:06:19.798440 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.790613 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 18:06:19.807437 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.799614 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" podUID="277a1fec-5c82-449d-839f-e843fa0f813d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:19.936097 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.936006 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 18:06:19.973421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.941541 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" podUID="aca6224c-f93a-4d51-b37e-f3368efa25dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 18:06:19.975601 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.975569 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 18:06:19.977985 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:19.977950 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" podUID="0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 18:06:20.081058 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:06:20.074006 5805 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.059s" Nov 03 18:06:20.081058 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.074091 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:06:20.081058 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.074150 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:06:20.081058 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.074236 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 18:06:20.081058 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.074271 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:06:20.146482 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.122746 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 18:06:20.146482 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.122801 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" podUID="e1392256-ac5e-4f9a-90da-87b002ebd848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 18:06:20.146482 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.130598 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" podUID="ec4a7156-46e6-4274-9794-e6d1f14aec3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:20.147003 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.146956 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:06:20.151255 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.151229 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 18:06:20.176602 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.176545 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" podUID="94ef80bb-de9b-4163-9db7-0fd6f19a8eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 18:06:20.176839 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.160640 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 18:06:20.176989 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.176967 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" podUID="2df51a82-bd64-4ad8-a164-d5af9ee53e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 18:06:20.369081 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.369033 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:20.488534 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.488489 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:20.568560 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.565805 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:20.568560 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.566138 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" podUID="e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 03 18:06:20.568560 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.566423 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" podUID="2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 18:06:20.632171 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.632077 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 18:06:20.681005 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.655020 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:20.682763 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.682735 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" podUID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:20.683188 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.682660 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" podUID="0b450130-5eea-4fe2-896c-62ba2fe84807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 18:06:20.807421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.789572 5805 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:20.825418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.808037 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 18:06:20.825962 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.814491 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" podUID="277a1fec-5c82-449d-839f-e843fa0f813d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 18:06:20.919475 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.896778 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 18:06:20.919475 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.896843 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" podUID="aca6224c-f93a-4d51-b37e-f3368efa25dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 18:06:20.919475 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.897096 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 18:06:20.919475 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:20.897124 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" podUID="0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 18:06:21.265428 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:21.190774 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" podUID="ec4a7156-46e6-4274-9794-e6d1f14aec3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:21.265428 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:21.192347 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 18:06:21.265428 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:21.192387 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" podUID="2df51a82-bd64-4ad8-a164-d5af9ee53e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 18:06:21.265428 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:21.194005 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 18:06:21.265428 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:21.194039 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" podUID="94ef80bb-de9b-4163-9db7-0fd6f19a8eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 18:06:21.267705 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:21.267663 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 18:06:21.267938 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:21.267894 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" podUID="e1392256-ac5e-4f9a-90da-87b002ebd848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 18:06:21.268148 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:21.268109 5805 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Nov 03 18:06:21.400421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:21.375757 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:21.519482 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:21.498536 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:21.551666 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:21.551628 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 18:06:21.552419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:21.552362 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" podUID="0b450130-5eea-4fe2-896c-62ba2fe84807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 18:06:21.688478 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:21.662344 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 03 18:06:21.688478 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:21.679917 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" podUID="e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 03 18:06:21.688478 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:21.680264 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" podUID="2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 18:06:21.694303 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:21.694263 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" podUID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:21.726742 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:21.694550 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:21.842479 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:21.825869 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:21.845196 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:21.845147 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" podUID="277a1fec-5c82-449d-839f-e843fa0f813d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:21.952422 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:21.934258 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 18:06:21.960419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:21.956488 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" podUID="0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 18:06:21.978641 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:21.943668 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 18:06:22.045544 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.045494 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" podUID="aca6224c-f93a-4d51-b37e-f3368efa25dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 18:06:22.229421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.223387 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" podUID="ec4a7156-46e6-4274-9794-e6d1f14aec3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:22.262421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.261526 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 18:06:22.262421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.261587 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" podUID="94ef80bb-de9b-4163-9db7-0fd6f19a8eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 18:06:22.291419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.261298 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 18:06:22.291419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.286535 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" podUID="2df51a82-bd64-4ad8-a164-d5af9ee53e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 18:06:22.405807 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.404480 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:22.439316 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.439278 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 18:06:22.467541 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.467491 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" podUID="e1392256-ac5e-4f9a-90da-87b002ebd848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 18:06:22.467780 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.452480 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 18:06:22.513671 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.513622 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 03 18:06:22.595339 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.594860 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 18:06:22.595339 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.594916 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" podUID="0b450130-5eea-4fe2-896c-62ba2fe84807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 18:06:22.645487 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.644330 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd8d1c6-xgqjv" event={"ID":"3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8","Type":"ContainerStarted","Data":"e1801f664c252265ca4db55fa3df725af5ad5f8ff3dd6fe0e33f72ee77e47605"} Nov 03 18:06:22.687442 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.683619 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" podUID="2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:22.687442 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.683959 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 03 18:06:22.759417 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.755699 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 03 18:06:22.812169 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.778328 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 18:06:22.831542 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.831490 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:22.852425 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.844660 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd8d1c6-xgqjv" podStartSLOduration=6.844640424 podStartE2EDuration="6.844640424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:22.818713903 +0000 UTC m=+1219.952342915" watchObservedRunningTime="2026-11-03 18:06:22.844640424 +0000 UTC m=+1219.978269460" Nov 03 18:06:22.852738 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.852702 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" podUID="277a1fec-5c82-449d-839f-e843fa0f813d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:22.911466 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.909987 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 18:06:22.911466 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.910048 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" podUID="0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 18:06:22.937601 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.937517 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 18:06:22.937841 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:22.937811 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" podUID="aca6224c-f93a-4d51-b37e-f3368efa25dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 18:06:23.155366 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:23.155331 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 18:06:23.274471 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:23.268425 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" podUID="94ef80bb-de9b-4163-9db7-0fd6f19a8eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 18:06:23.274471 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:23.238914 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" podUID="ec4a7156-46e6-4274-9794-e6d1f14aec3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:23.274471 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:23.155636 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 18:06:23.274471 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:23.268533 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" podUID="e1392256-ac5e-4f9a-90da-87b002ebd848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 18:06:23.318425 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:23.310532 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 18:06:23.318425 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:23.312573 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 18:06:23.318425 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:23.312623 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" podUID="2df51a82-bd64-4ad8-a164-d5af9ee53e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 18:06:23.441417 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:23.414493 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:23.580794 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:23.546738 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Nov 03 18:06:23.580794 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:23.546795 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" podUID="0b450130-5eea-4fe2-896c-62ba2fe84807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Nov 03 18:06:23.768437 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:23.723640 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" podUID="2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 18:06:23.768437 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:23.766643 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" podUID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:23.866472 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:23.837715 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:23.866472 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:23.839111 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 18:06:23.866472 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:23.854505 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" podUID="277a1fec-5c82-449d-839f-e843fa0f813d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 18:06:23.897049 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:23.897002 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 18:06:23.897328 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:23.897278 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" podUID="0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 18:06:23.897787 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:23.897721 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 18:06:23.897963 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:23.897943 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" podUID="aca6224c-f93a-4d51-b37e-f3368efa25dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 18:06:23.908564 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:23.899245 5805 generic.go:356] "Generic (PLEG): container finished" podID="3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8" containerID="e1801f664c252265ca4db55fa3df725af5ad5f8ff3dd6fe0e33f72ee77e47605" exitCode=0 Nov 03 18:06:23.908903 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:23.899269 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd8d1c6-xgqjv" event={"ID":"3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8","Type":"ContainerDied","Data":"e1801f664c252265ca4db55fa3df725af5ad5f8ff3dd6fe0e33f72ee77e47605"} Nov 03 18:06:23.909252 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:23.909225 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:23.994706 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:23.994674 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:06:24.115894 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:24.115860 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 18:06:24.118475 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:24.118388 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" podUID="e1392256-ac5e-4f9a-90da-87b002ebd848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 18:06:24.148159 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:24.148127 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 18:06:24.148437 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:24.148407 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" podUID="94ef80bb-de9b-4163-9db7-0fd6f19a8eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 18:06:24.176774 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:24.176739 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 18:06:24.177071 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:24.177037 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" podUID="2df51a82-bd64-4ad8-a164-d5af9ee53e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 18:06:24.316420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:24.315178 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 03 18:06:24.423643 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:24.423539 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:24.729547 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:24.723252 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:24.775014 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:24.774968 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" podUID="2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:24.801418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:24.775250 5805 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" podUID="277a1fec-5c82-449d-839f-e843fa0f813d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded" Nov 03 18:06:24.801418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:24.776464 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" podUID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:24.819878 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:24.819819 5805 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-46sr7" podUID="01ee4c19-32e1-433a-88b6-d6f6209bae7f" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.23:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:24.843474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:24.842610 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:24.843767 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:24.843744 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 18:06:24.855532 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:24.855490 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" podUID="277a1fec-5c82-449d-839f-e843fa0f813d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:24.882545 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:24.882496 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:25.005654 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:25.005554 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:25.243422 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:25.229337 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 18:06:25.243422 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:25.229438 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" podUID="e1392256-ac5e-4f9a-90da-87b002ebd848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 18:06:25.325484 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:25.293635 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 18:06:25.325484 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:25.293704 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" podUID="94ef80bb-de9b-4163-9db7-0fd6f19a8eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 18:06:25.325484 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:25.293819 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Nov 03 18:06:25.325484 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:25.293844 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" podUID="2df51a82-bd64-4ad8-a164-d5af9ee53e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Nov 03 18:06:25.325484 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:25.301205 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" podUID="ec4a7156-46e6-4274-9794-e6d1f14aec3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 18:06:25.353420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:25.336833 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:25.448590 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:25.448546 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:25.580505 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:25.559192 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 18:06:25.666642 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:25.666605 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:25.758620 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:25.758121 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" podUID="e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:25.854485 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:25.760162 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:25.854799 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:25.809627 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" podUID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 03 18:06:25.855109 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:25.843168 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" podUID="2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:25.873518 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:25.865322 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:25.898615 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:25.868195 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" podUID="277a1fec-5c82-449d-839f-e843fa0f813d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:25.898999 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:25.886740 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:25.899306 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:25.893580 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:06:25.899554 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:25.899502 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" podUID="0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:26.096421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.084821 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:26.120473 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.097461 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:26.120808 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.120788 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:06:26.121032 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.121020 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:06:26.121156 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.111505 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 18:06:26.124507 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.124474 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" podUID="e1392256-ac5e-4f9a-90da-87b002ebd848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 18:06:26.136416 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.109789 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:26.136880 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.108951 5805 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded" Nov 03 18:06:26.145426 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.137958 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 18:06:26.145426 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.138011 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" podUID="94ef80bb-de9b-4163-9db7-0fd6f19a8eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 18:06:26.159426 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.149609 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:06:26.159426 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.149678 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" podUID="aca6224c-f93a-4d51-b37e-f3368efa25dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:26.267430 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.265418 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:26.269688 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.269660 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:26.306443 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.305508 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" podUID="ec4a7156-46e6-4274-9794-e6d1f14aec3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:26.306443 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.305976 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:06:26.337563 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.337496 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:26.364731 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.364701 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:06:26.570750 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.569280 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:06:26.570750 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.570207 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 18:06:26.617033 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:06:26.616994 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:06:26.623909 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.622990 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:06:26.659021 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.658989 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 18:06:26.708445 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.708069 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:26.710573 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.709612 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:26.732255 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.725754 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:06:26.744927 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.735140 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:06:26.787906 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.781234 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:06:26.787906 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.785121 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 18:06:26.787906 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.787185 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 18:06:26.789653 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.789630 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 18:06:26.815461 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.812678 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs"] Nov 03 18:06:26.818283 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.818262 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:26.849132 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.843563 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd8d1c6-xgqjv" Nov 03 18:06:26.851533 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.849533 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:06:26.861358 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.861331 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs"] Nov 03 18:06:26.935317 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.935278 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-workload-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:26.935499 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.935332 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-credential-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:26.935499 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.935378 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-data\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:26.935883 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.935855 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istiod-ca-cert\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:26.935958 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.935941 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s27bw\" (UniqueName: \"kubernetes.io/projected/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-kube-api-access-s27bw\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:26.936129 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.936098 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-envoy\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:26.936270 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.936252 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-podinfo\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:26.936307 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.936286 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-token\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:26.936340 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:26.936315 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-workload-certs\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:27.037322 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.037287 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8-maas-api-xks-ca\") pod \"3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8\" (UID: \"3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8\") " Nov 03 18:06:27.037514 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.037368 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8-maas-api-openshift-ca\") pod \"3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8\" (UID: \"3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8\") " Nov 03 18:06:27.037583 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.037526 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8-kube-api-access-c28hc\" (UniqueName: \"kubernetes.io/projected/3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8-kube-api-access-c28hc\") pod \"3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8\" (UID: \"3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8\") " Nov 03 18:06:27.037845 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.037808 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istiod-ca-cert\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:27.037963 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.037851 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s27bw\" (UniqueName: \"kubernetes.io/projected/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-kube-api-access-s27bw\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:27.037963 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.037899 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8-maas-api-openshift-ca" pod "3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8" (UID: "3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:06:27.037963 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.037906 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-envoy\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:27.038167 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.038046 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-podinfo\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:27.038167 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.038078 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-token\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:27.038280 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.038184 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-workload-certs\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:27.038348 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.038281 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-workload-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:27.038348 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.038317 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-credential-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:27.038503 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.038358 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-data\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:27.038661 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.038638 5805 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8-maas-api-openshift-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:06:27.038974 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.038947 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-workload-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:27.039197 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.039171 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istiod-ca-cert\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:27.039280 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.039237 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-workload-certs\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:27.039746 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.039684 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-data\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:27.039916 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.039876 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-credential-socket\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:27.041354 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.041330 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-envoy\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:27.041672 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.041645 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8-kube-api-access-c28hc" pod "3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8" (UID: "3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8"). InnerVolumeSpecName "kube-api-access-c28hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:06:27.041831 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.041804 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8-maas-api-xks-ca" pod "3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8" (UID: "3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:06:27.041932 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.041847 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-podinfo\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:27.115604 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.115474 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s27bw\" (UniqueName: \"kubernetes.io/projected/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-kube-api-access-s27bw\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:27.116330 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.115008 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-token\") pod \"e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:27.116330 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.116222 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 18:06:27.116330 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.116278 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" podUID="e1392256-ac5e-4f9a-90da-87b002ebd848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 18:06:27.135023 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.134988 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 18:06:27.135156 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.135047 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" podUID="94ef80bb-de9b-4163-9db7-0fd6f19a8eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 18:06:27.139828 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.139807 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c28hc\" (UniqueName: \"kubernetes.io/projected/3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8-kube-api-access-c28hc\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:06:27.139899 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.139831 5805 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8-maas-api-xks-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:06:27.168312 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.168289 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:27.630737 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.630702 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs"] Nov 03 18:06:27.642896 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.642856 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd8d1c6-xgqjv" event={"ID":"3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8","Type":"ContainerDied","Data":"8af86d88ba7c92980e7131329212b0b5a69f3adefe863ac70f9a6e670c77c633"} Nov 03 18:06:27.643023 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.642913 5805 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8af86d88ba7c92980e7131329212b0b5a69f3adefe863ac70f9a6e670c77c633" Nov 03 18:06:27.643874 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.643720 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd8d1c6-xgqjv" Nov 03 18:06:27.645548 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:06:27.645522 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5efde3b1_24ad_4da6_81e9_09c7c7aa181c.slice/crio-962f99a87724695a549e29cfae109ac2fde071c0b5b75b12467fef6dc7e975b4 WatchSource:0}: Error finding container 962f99a87724695a549e29cfae109ac2fde071c0b5b75b12467fef6dc7e975b4: Status 404 returned error can't find the container with id 962f99a87724695a549e29cfae109ac2fde071c0b5b75b12467fef6dc7e975b4 Nov 03 18:06:27.648718 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.648684 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:06:27.648913 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.648890 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:06:27.649081 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:27.649060 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:06:28.149729 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:28.149695 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:28.354413 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:28.349692 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:28.354413 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:28.350727 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:28.354413 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:28.351897 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:06:28.355250 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:28.355221 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:28.416016 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:28.415720 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:28.417414 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:28.417056 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:28.443442 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:28.420351 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:06:28.654889 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:28.654853 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" event={"ID":"5efde3b1-24ad-4da6-81e9-09c7c7aa181c","Type":"ContainerStarted","Data":"bf16ff46d54088f2855b6aeaf974c8d54195ae79252d3bd17cf036c8dc1596d9"} Nov 03 18:06:28.655031 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:28.654916 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" event={"ID":"5efde3b1-24ad-4da6-81e9-09c7c7aa181c","Type":"ContainerStarted","Data":"962f99a87724695a549e29cfae109ac2fde071c0b5b75b12467fef6dc7e975b4"} Nov 03 18:06:28.842419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:28.829820 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" podStartSLOduration=2.8297965400000002 podStartE2EDuration="2.82979654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:28.822159965 +0000 UTC m=+1225.955788978" watchObservedRunningTime="2026-11-03 18:06:28.82979654 +0000 UTC m=+1225.963425551" Nov 03 18:06:28.961964 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:28.961931 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:29.168679 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:29.168589 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:29.170490 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:29.170447 5805 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 18:06:29.170622 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:29.170502 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" podUID="5efde3b1-24ad-4da6-81e9-09c7c7aa181c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 18:06:29.596308 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:29.596269 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:30.168673 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:30.168639 5805 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 18:06:30.169051 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:30.168693 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" podUID="5efde3b1-24ad-4da6-81e9-09c7c7aa181c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 18:06:31.168786 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:31.168754 5805 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 18:06:31.169265 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:31.168815 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" podUID="5efde3b1-24ad-4da6-81e9-09c7c7aa181c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 18:06:32.168879 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:32.168843 5805 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 18:06:32.169355 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:32.168891 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" podUID="5efde3b1-24ad-4da6-81e9-09c7c7aa181c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 18:06:33.168739 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:33.168661 5805 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 18:06:33.168739 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:33.168711 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" podUID="5efde3b1-24ad-4da6-81e9-09c7c7aa181c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 18:06:33.340417 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:33.330955 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:33.543468 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:33.541811 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:34.169840 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:34.169789 5805 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 18:06:34.169994 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:34.169879 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" podUID="5efde3b1-24ad-4da6-81e9-09c7c7aa181c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 18:06:34.275836 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:34.275800 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:35.173439 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:35.173387 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:35.173948 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:35.173925 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:35.174552 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:35.174535 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:06:35.418177 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:35.418084 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:35.621281 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:35.621245 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:35.817376 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:35.817340 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:36.046945 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:36.046867 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:36.708532 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:36.708498 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:37.039562 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:37.039458 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:38.975277 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:38.975239 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:39.204314 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:39.204277 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:40.040431 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:40.040383 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:40.254375 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:40.254279 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:40.640546 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:40.640519 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:40.844813 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:40.844764 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:43.697412 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:43.697368 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:43.962063 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:43.961990 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:44.267133 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:44.267037 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:47.592418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:47.567003 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" podUID="e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 03 18:06:47.698373 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:47.698320 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:47.699063 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:47.698642 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:47.700226 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:47.698671 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" podUID="2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:47.727508 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:47.727468 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:47.875474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:47.875039 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" podUID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:47.875474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:47.875115 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" podUID="277a1fec-5c82-449d-839f-e843fa0f813d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:47.875474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:47.875459 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" podUID="ec4a7156-46e6-4274-9794-e6d1f14aec3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 18:06:48.006420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:48.005711 5805 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" podUID="e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:48.018419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:48.012517 5805 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:48.170586 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:48.170482 5805 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:48.251739 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:48.251695 5805 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" podUID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:48.353541 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:48.353483 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 03 18:06:48.410688 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:48.410630 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:48.717431 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:48.709476 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:48.717431 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:48.710149 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 18:06:48.717431 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:48.710831 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" podUID="2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:48.717431 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:48.711020 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:06:48.732620 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:48.732555 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:48.733253 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:48.733234 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:06:48.797856 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:48.797696 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" podUID="e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:48.807751 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:48.807530 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 18:06:48.893461 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:48.879045 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" podUID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:48.893461 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:48.879448 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 18:06:48.893461 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:48.880795 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" podUID="ec4a7156-46e6-4274-9794-e6d1f14aec3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:48.893461 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:48.881058 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" podUID="277a1fec-5c82-449d-839f-e843fa0f813d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 18:06:48.893461 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:48.881330 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:06:48.893461 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:48.881381 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:06:49.331619 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:49.331240 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:06:49.331619 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:49.331309 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" podUID="f403b27f-184b-42fd-839a-92fa3b4cdc2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:49.453641 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:49.453583 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:49.478853 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:49.478823 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 18:06:49.725463 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:49.721085 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:49.738441 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:49.735691 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:49.738441 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:49.737464 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" podUID="2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:49.759505 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:49.759466 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 03 18:06:49.792039 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:49.777791 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:06:49.792273 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:49.792244 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" podUID="10efc538-5a7a-40ce-b6be-819a2cddbec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:49.828866 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:49.802010 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" podUID="e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 03 18:06:49.883419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:49.879337 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" podUID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:49.891586 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:49.891550 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" podUID="ec4a7156-46e6-4274-9794-e6d1f14aec3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 18:06:49.900493 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:49.891500 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" podUID="277a1fec-5c82-449d-839f-e843fa0f813d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:49.985598 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:49.985517 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:06:50.034694 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:50.034661 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 18:06:50.481520 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:50.481473 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:50.552416 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:50.541882 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 18:06:50.552416 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:50.543083 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:06:50.552416 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:50.543131 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:06:50.552416 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:50.543164 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:06:50.576670 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:50.575546 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 18:06:50.579803 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:50.579783 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 18:06:50.679835 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:50.679801 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:54.411552 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:54.411518 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z"] Nov 03 18:06:54.412346 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:54.412315 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" podUID="f8c9e705-3831-499a-9ff4-3b03aeac8617" containerName="istio-proxy" containerID="cri-o://1089f0d13fee4c34da914ac3ae10a87ff4d9c039e02dbd63cbf50aefc6f544db" gracePeriod=30 Nov 03 18:06:54.977852 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:54.977804 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:55.182177 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:55.182134 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:55.381276 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:55.381239 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:55.634842 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:55.634768 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:56.236132 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:06:56.236093 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:06:57.553153 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:57.553110 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:58.341264 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:58.341226 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:58.620178 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:58.620096 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:59.709917 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.709850 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:06:59.758384 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.758343 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:06:59.869677 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.869647 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8c9e705-3831-499a-9ff4-3b03aeac8617-kube-api-access-8kgwj\" (UniqueName: \"kubernetes.io/projected/f8c9e705-3831-499a-9ff4-3b03aeac8617-kube-api-access-8kgwj\") pod \"f8c9e705-3831-499a-9ff4-3b03aeac8617\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " Nov 03 18:06:59.869950 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.869920 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-token\" (UniqueName: \"kubernetes.io/projected/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-token\") pod \"f8c9e705-3831-499a-9ff4-3b03aeac8617\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " Nov 03 18:06:59.870077 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.869961 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-workload-certs\") pod \"f8c9e705-3831-499a-9ff4-3b03aeac8617\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " Nov 03 18:06:59.870146 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.870097 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8c9e705-3831-499a-9ff4-3b03aeac8617-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c9e705-3831-499a-9ff4-3b03aeac8617-istiod-ca-cert\") pod \"f8c9e705-3831-499a-9ff4-3b03aeac8617\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " Nov 03 18:06:59.870209 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.870186 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-podinfo\") pod \"f8c9e705-3831-499a-9ff4-3b03aeac8617\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " Nov 03 18:06:59.870266 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.870229 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-envoy\") pod \"f8c9e705-3831-499a-9ff4-3b03aeac8617\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " Nov 03 18:06:59.870322 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.870305 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-credential-socket\") pod \"f8c9e705-3831-499a-9ff4-3b03aeac8617\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " Nov 03 18:06:59.870452 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.870420 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-workload-socket\") pod \"f8c9e705-3831-499a-9ff4-3b03aeac8617\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " Nov 03 18:06:59.870606 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.870578 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8c9e705-3831-499a-9ff4-3b03aeac8617-istiod-ca-cert" pod "f8c9e705-3831-499a-9ff4-3b03aeac8617" (UID: "f8c9e705-3831-499a-9ff4-3b03aeac8617"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:06:59.870663 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.870617 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-workload-certs" pod "f8c9e705-3831-499a-9ff4-3b03aeac8617" (UID: "f8c9e705-3831-499a-9ff4-3b03aeac8617"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:06:59.870796 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.870760 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-data\") pod \"f8c9e705-3831-499a-9ff4-3b03aeac8617\" (UID: \"f8c9e705-3831-499a-9ff4-3b03aeac8617\") " Nov 03 18:06:59.871346 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.871327 5805 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-workload-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:06:59.871498 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.871477 5805 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c9e705-3831-499a-9ff4-3b03aeac8617-istiod-ca-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:06:59.871498 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.871324 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-workload-socket" pod "f8c9e705-3831-499a-9ff4-3b03aeac8617" (UID: "f8c9e705-3831-499a-9ff4-3b03aeac8617"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:06:59.871702 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.871366 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-credential-socket" pod "f8c9e705-3831-499a-9ff4-3b03aeac8617" (UID: "f8c9e705-3831-499a-9ff4-3b03aeac8617"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:06:59.871702 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.871547 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-data" pod "f8c9e705-3831-499a-9ff4-3b03aeac8617" (UID: "f8c9e705-3831-499a-9ff4-3b03aeac8617"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:06:59.875694 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.875644 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c9e705-3831-499a-9ff4-3b03aeac8617-kube-api-access-8kgwj" pod "f8c9e705-3831-499a-9ff4-3b03aeac8617" (UID: "f8c9e705-3831-499a-9ff4-3b03aeac8617"). InnerVolumeSpecName "kube-api-access-8kgwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:06:59.877409 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.876098 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-podinfo" pod "f8c9e705-3831-499a-9ff4-3b03aeac8617" (UID: "f8c9e705-3831-499a-9ff4-3b03aeac8617"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:06:59.877758 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.877733 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-envoy" pod "f8c9e705-3831-499a-9ff4-3b03aeac8617" (UID: "f8c9e705-3831-499a-9ff4-3b03aeac8617"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:06:59.881272 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.880903 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-token" pod "f8c9e705-3831-499a-9ff4-3b03aeac8617" (UID: "f8c9e705-3831-499a-9ff4-3b03aeac8617"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:06:59.972635 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.972524 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kgwj\" (UniqueName: \"kubernetes.io/projected/f8c9e705-3831-499a-9ff4-3b03aeac8617-kube-api-access-8kgwj\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:06:59.972635 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.972564 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:06:59.972635 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.972582 5805 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-podinfo\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:06:59.972635 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.972596 5805 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-envoy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:06:59.972635 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.972610 5805 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-credential-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:06:59.972635 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.972624 5805 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-workload-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:06:59.972635 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:06:59.972638 5805 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c9e705-3831-499a-9ff4-3b03aeac8617-istio-data\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:07:00.089667 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:00.089625 5805 generic.go:356] "Generic (PLEG): container finished" podID="f8c9e705-3831-499a-9ff4-3b03aeac8617" containerID="1089f0d13fee4c34da914ac3ae10a87ff4d9c039e02dbd63cbf50aefc6f544db" exitCode=0 Nov 03 18:07:00.089835 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:00.089718 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" Nov 03 18:07:00.089835 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:00.089777 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" event={"ID":"f8c9e705-3831-499a-9ff4-3b03aeac8617","Type":"ContainerDied","Data":"1089f0d13fee4c34da914ac3ae10a87ff4d9c039e02dbd63cbf50aefc6f544db"} Nov 03 18:07:00.089835 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:00.089821 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z" event={"ID":"f8c9e705-3831-499a-9ff4-3b03aeac8617","Type":"ContainerDied","Data":"7c36578acfe3a9861309eab6896d8c05283d3f2938b0399d6328235f0900ebce"} Nov 03 18:07:00.089943 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:00.089840 5805 scope.go:122] "RemoveContainer" containerID="1089f0d13fee4c34da914ac3ae10a87ff4d9c039e02dbd63cbf50aefc6f544db" Nov 03 18:07:00.100954 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:00.100899 5805 scope.go:122] "RemoveContainer" containerID="1089f0d13fee4c34da914ac3ae10a87ff4d9c039e02dbd63cbf50aefc6f544db" Nov 03 18:07:00.101315 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:07:00.101290 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1089f0d13fee4c34da914ac3ae10a87ff4d9c039e02dbd63cbf50aefc6f544db\": container with ID starting with 1089f0d13fee4c34da914ac3ae10a87ff4d9c039e02dbd63cbf50aefc6f544db not found: ID does not exist" containerID="1089f0d13fee4c34da914ac3ae10a87ff4d9c039e02dbd63cbf50aefc6f544db" Nov 03 18:07:00.101442 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:00.101323 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1089f0d13fee4c34da914ac3ae10a87ff4d9c039e02dbd63cbf50aefc6f544db"} err="failed to get container status \"1089f0d13fee4c34da914ac3ae10a87ff4d9c039e02dbd63cbf50aefc6f544db\": rpc error: code = NotFound desc = could not find container \"1089f0d13fee4c34da914ac3ae10a87ff4d9c039e02dbd63cbf50aefc6f544db\": container with ID starting with 1089f0d13fee4c34da914ac3ae10a87ff4d9c039e02dbd63cbf50aefc6f544db not found: ID does not exist" Nov 03 18:07:00.234563 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:00.234486 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:00.245140 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:00.245113 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z"] Nov 03 18:07:00.257599 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:00.257572 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-zjg5z"] Nov 03 18:07:00.429558 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:00.429526 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:00.916089 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:00.916057 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh"] Nov 03 18:07:00.916816 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:00.916796 5805 cpu_manager.go:414] "containerMap: removing container" podUID="3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8" containerName="revoke-keys" Nov 03 18:07:00.916816 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:00.916815 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8" containerName="revoke-keys" Nov 03 18:07:00.916967 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:00.916836 5805 cpu_manager.go:414] "containerMap: removing container" podUID="f8c9e705-3831-499a-9ff4-3b03aeac8617" containerName="istio-proxy" Nov 03 18:07:00.916967 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:00.916841 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8c9e705-3831-499a-9ff4-3b03aeac8617" containerName="istio-proxy" Nov 03 18:07:00.916967 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:00.916961 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8" containerName="revoke-keys" Nov 03 18:07:00.917124 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:00.916975 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8c9e705-3831-499a-9ff4-3b03aeac8617" containerName="istio-proxy" Nov 03 18:07:00.921710 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:00.921686 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh" Nov 03 18:07:00.970682 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:00.970646 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-29bb4e29\"" Nov 03 18:07:00.972545 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:00.972517 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-29bb4e29-dockercfg-s8kqs\"" Nov 03 18:07:00.980184 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:00.980153 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8c9e705-3831-499a-9ff4-3b03aeac8617" path="/var/lib/kubelet/pods/f8c9e705-3831-499a-9ff4-3b03aeac8617/volumes" Nov 03 18:07:00.980623 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:00.980598 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh"] Nov 03 18:07:01.089310 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.089275 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brwz4\" (UniqueName: \"kubernetes.io/projected/e554043b-309f-4fa8-ae38-77211123fb58-kube-api-access-brwz4\") pod \"payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh\" (UID: \"e554043b-309f-4fa8-ae38-77211123fb58\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh" Nov 03 18:07:01.089510 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.089360 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e554043b-309f-4fa8-ae38-77211123fb58-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh\" (UID: \"e554043b-309f-4fa8-ae38-77211123fb58\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh" Nov 03 18:07:01.191157 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.191065 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brwz4\" (UniqueName: \"kubernetes.io/projected/e554043b-309f-4fa8-ae38-77211123fb58-kube-api-access-brwz4\") pod \"payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh\" (UID: \"e554043b-309f-4fa8-ae38-77211123fb58\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh" Nov 03 18:07:01.191157 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.191138 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e554043b-309f-4fa8-ae38-77211123fb58-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh\" (UID: \"e554043b-309f-4fa8-ae38-77211123fb58\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh" Nov 03 18:07:01.192047 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.192022 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e554043b-309f-4fa8-ae38-77211123fb58-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh\" (UID: \"e554043b-309f-4fa8-ae38-77211123fb58\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh" Nov 03 18:07:01.221734 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.221682 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n"] Nov 03 18:07:01.226274 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.226250 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:01.248437 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.248390 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brwz4\" (UniqueName: \"kubernetes.io/projected/e554043b-309f-4fa8-ae38-77211123fb58-kube-api-access-brwz4\") pod \"payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh\" (UID: \"e554043b-309f-4fa8-ae38-77211123fb58\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh" Nov 03 18:07:01.277528 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.277503 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 18:07:01.277528 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.277516 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 18:07:01.277705 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.277519 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 18:07:01.277705 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.277638 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xxxzp\"" Nov 03 18:07:01.284594 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.284562 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n"] Nov 03 18:07:01.394158 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.394122 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mp9n\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:01.394321 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.394170 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mp9n\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:01.394321 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.394297 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mp9n\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:01.394434 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.394350 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sbf4\" (UniqueName: \"kubernetes.io/projected/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-kube-api-access-5sbf4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mp9n\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:01.394434 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.394371 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mp9n\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:01.394564 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.394538 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mp9n\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:01.394606 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.394580 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mp9n\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:01.495488 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.495408 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mp9n\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:01.495488 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.495465 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sbf4\" (UniqueName: \"kubernetes.io/projected/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-kube-api-access-5sbf4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mp9n\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:01.495488 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.495485 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mp9n\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:01.495729 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.495533 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mp9n\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:01.495729 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.495557 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mp9n\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:01.495729 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.495675 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mp9n\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:01.495829 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.495726 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mp9n\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:01.496424 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.496381 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mp9n\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:01.498157 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.498128 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mp9n\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:01.498282 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.498134 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mp9n\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:01.498282 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.498220 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mp9n\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:01.498389 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.498371 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mp9n\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:01.538563 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.538528 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh" Nov 03 18:07:01.552885 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.552853 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sbf4\" (UniqueName: \"kubernetes.io/projected/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-kube-api-access-5sbf4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mp9n\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:01.552885 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.552873 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9mp9n\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:01.810456 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.810427 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:01.839271 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.839241 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:01.927106 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:01.927074 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh"] Nov 03 18:07:01.935227 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:07:01.935181 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode554043b_309f_4fa8_ae38_77211123fb58.slice/crio-833261d87b30426de444de2efb053e855a40338a25ca16649021e46f3eda38cf WatchSource:0}: Error finding container 833261d87b30426de444de2efb053e855a40338a25ca16649021e46f3eda38cf: Status 404 returned error can't find the container with id 833261d87b30426de444de2efb053e855a40338a25ca16649021e46f3eda38cf Nov 03 18:07:02.026077 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:02.026045 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:02.124139 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:02.121954 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh" event={"ID":"e554043b-309f-4fa8-ae38-77211123fb58","Type":"ContainerStarted","Data":"38d53ac9a0e78be1f1de2defe7debc487debdd97988c168df51bf011f65d9bb4"} Nov 03 18:07:02.124139 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:02.122025 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh" event={"ID":"e554043b-309f-4fa8-ae38-77211123fb58","Type":"ContainerStarted","Data":"833261d87b30426de444de2efb053e855a40338a25ca16649021e46f3eda38cf"} Nov 03 18:07:02.124139 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:02.123141 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh" Nov 03 18:07:02.234756 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:02.234694 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh" podStartSLOduration=2.234679358 podStartE2EDuration="2.234679358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:02.232964876 +0000 UTC m=+1259.366593905" watchObservedRunningTime="2026-11-03 18:07:02.234679358 +0000 UTC m=+1259.368308370" Nov 03 18:07:02.250003 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:02.249958 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n"] Nov 03 18:07:02.287264 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:07:02.287230 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe4928a_7b15_472b_8c9e_fc28b4c46e43.slice/crio-b4d2591c6e3d09c2f5452cb882bb9991268c72698d1181ecb26901b5b335e058 WatchSource:0}: Error finding container b4d2591c6e3d09c2f5452cb882bb9991268c72698d1181ecb26901b5b335e058: Status 404 returned error can't find the container with id b4d2591c6e3d09c2f5452cb882bb9991268c72698d1181ecb26901b5b335e058 Nov 03 18:07:02.289908 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:02.289882 5805 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:07:02.925371 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:02.925340 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:03.129600 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:03.129564 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" event={"ID":"0fe4928a-7b15-472b-8c9e-fc28b4c46e43","Type":"ContainerStarted","Data":"b4d2591c6e3d09c2f5452cb882bb9991268c72698d1181ecb26901b5b335e058"} Nov 03 18:07:04.281656 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:04.281617 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:04.872731 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:04.872680 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:05.073815 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:05.073778 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:05.377460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:05.376918 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:06.974440 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:06.974383 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:07:06.974758 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:06.974483 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:07:07.167734 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:07.167681 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" event={"ID":"0fe4928a-7b15-472b-8c9e-fc28b4c46e43","Type":"ContainerStarted","Data":"40d4b3d869ec85e486371787e979ad7624784a528175a64df25f149c9540d7dd"} Nov 03 18:07:07.167939 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:07.167751 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:07.374139 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:07.374086 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" podStartSLOduration=1.6915012200000001 podStartE2EDuration="6.374072301s" totalImagesPullingTime="4.682571081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:07.327574861 +0000 UTC m=+1264.461203904" watchObservedRunningTime="2026-11-03 18:07:07.374072301 +0000 UTC m=+1264.507701310" Nov 03 18:07:08.178348 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:08.178303 5805 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-9mp9n container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:07:08.178831 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:08.178383 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" podUID="0fe4928a-7b15-472b-8c9e-fc28b4c46e43" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:07:08.217701 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:08.217666 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:08.432705 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:08.432623 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:09.077351 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:09.077311 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:09.278843 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:09.278811 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:09.786457 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:09.786384 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:11.176620 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:11.176589 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:07:12.251917 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:12.251826 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:12.453567 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:12.453532 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:13.322787 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.322756 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4"] Nov 03 18:07:13.327205 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.327181 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.384551 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.384526 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.384697 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.384584 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44286112-623b-4654-9b84-7eed4dd067d9-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.384750 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.384707 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44286112-623b-4654-9b84-7eed4dd067d9-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.384750 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.384740 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.384827 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.384781 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndc28\" (UniqueName: \"kubernetes.io/projected/44286112-623b-4654-9b84-7eed4dd067d9-kube-api-access-ndc28\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.384913 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.384890 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.384960 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.384934 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44286112-623b-4654-9b84-7eed4dd067d9-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.384960 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.384953 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.385053 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.385040 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.427889 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.427858 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4"] Nov 03 18:07:13.486195 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.486153 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.486654 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.486244 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44286112-623b-4654-9b84-7eed4dd067d9-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.486654 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.486312 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44286112-623b-4654-9b84-7eed4dd067d9-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.486654 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.486362 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.486654 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.486437 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndc28\" (UniqueName: \"kubernetes.io/projected/44286112-623b-4654-9b84-7eed4dd067d9-kube-api-access-ndc28\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.486654 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.486476 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.486654 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.486507 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44286112-623b-4654-9b84-7eed4dd067d9-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.486654 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.486533 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.486654 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.486578 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.486654 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.486635 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.487151 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.486933 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.487519 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.487485 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.487684 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.487663 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44286112-623b-4654-9b84-7eed4dd067d9-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.487755 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.487671 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.489371 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.489347 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.489469 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.489452 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44286112-623b-4654-9b84-7eed4dd067d9-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.543603 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.543576 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44286112-623b-4654-9b84-7eed4dd067d9-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.543724 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.543706 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndc28\" (UniqueName: \"kubernetes.io/projected/44286112-623b-4654-9b84-7eed4dd067d9-kube-api-access-ndc28\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.641545 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.641482 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:13.798458 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.798427 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw"] Nov 03 18:07:13.804138 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.804116 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.830560 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.830532 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7"] Nov 03 18:07:13.834732 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.834712 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.861109 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.861078 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw"] Nov 03 18:07:13.891761 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.891684 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.891909 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.891770 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d997ac9-3961-424f-88c5-7b2c2168eab9-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.891909 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.891809 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.891909 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.891842 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.892136 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.891918 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.892136 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.891948 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a819308-670a-4a83-8512-630337c61d5a-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.892136 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.891997 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a819308-670a-4a83-8512-630337c61d5a-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.892136 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.892026 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.892136 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.892085 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsl85\" (UniqueName: \"kubernetes.io/projected/7a819308-670a-4a83-8512-630337c61d5a-kube-api-access-vsl85\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.893915 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.892147 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a819308-670a-4a83-8512-630337c61d5a-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.893915 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.892212 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.893915 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.892236 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.893915 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.892263 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.893915 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.892317 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xznk\" (UniqueName: \"kubernetes.io/projected/7d997ac9-3961-424f-88c5-7b2c2168eab9-kube-api-access-6xznk\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.893915 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.892342 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.893915 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.892490 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.893915 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.892524 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.893915 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.892567 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.893915 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.892643 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7"] Nov 03 18:07:13.993715 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.993681 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.993715 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.993723 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a819308-670a-4a83-8512-630337c61d5a-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.993956 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.993755 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a819308-670a-4a83-8512-630337c61d5a-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.993956 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.993773 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.993956 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.993813 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsl85\" (UniqueName: \"kubernetes.io/projected/7a819308-670a-4a83-8512-630337c61d5a-kube-api-access-vsl85\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.993956 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.993867 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a819308-670a-4a83-8512-630337c61d5a-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.993956 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.993920 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.993956 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.993947 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.994262 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.994064 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.994262 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.994123 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xznk\" (UniqueName: \"kubernetes.io/projected/7d997ac9-3961-424f-88c5-7b2c2168eab9-kube-api-access-6xznk\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.994262 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.994153 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.994262 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.994206 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.994262 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.994245 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.994541 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.994282 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.994541 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.994307 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.994541 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.994323 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.994541 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.994366 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.994541 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.994440 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.994541 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.994509 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d997ac9-3961-424f-88c5-7b2c2168eab9-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.994844 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.994546 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.994844 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.994577 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.994844 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.994680 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.994844 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.994704 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a819308-670a-4a83-8512-630337c61d5a-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.995096 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.995071 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-credential-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.995361 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.995332 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-workload-socket\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.995483 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.995336 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-workload-certs\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.995483 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.995435 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d997ac9-3961-424f-88c5-7b2c2168eab9-istiod-ca-cert\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.995807 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.995778 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-data\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.996842 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.996800 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a819308-670a-4a83-8512-630337c61d5a-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.996962 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.996918 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-podinfo\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:13.997058 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.997036 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:13.997327 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:13.997306 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-envoy\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:14.052985 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:14.052951 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:14.053148 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:14.052961 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a819308-670a-4a83-8512-630337c61d5a-istio-token\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:14.053148 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:14.053062 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsl85\" (UniqueName: \"kubernetes.io/projected/7a819308-670a-4a83-8512-630337c61d5a-kube-api-access-vsl85\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:14.053226 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:14.053151 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xznk\" (UniqueName: \"kubernetes.io/projected/7d997ac9-3961-424f-88c5-7b2c2168eab9-kube-api-access-6xznk\") pod \"e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:14.073560 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:14.073532 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4"] Nov 03 18:07:14.078883 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:07:14.078855 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44286112_623b_4654_9b84_7eed4dd067d9.slice/crio-a3b18099a3eaf82b995f2e533a30c0dabddda68b79f021beab499bc3af514263 WatchSource:0}: Error finding container a3b18099a3eaf82b995f2e533a30c0dabddda68b79f021beab499bc3af514263: Status 404 returned error can't find the container with id a3b18099a3eaf82b995f2e533a30c0dabddda68b79f021beab499bc3af514263 Nov 03 18:07:14.081545 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:14.081516 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:07:14.081628 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:14.081583 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:07:14.081628 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:14.081612 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:07:14.122005 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:14.121981 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:14.140168 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:14.140149 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh" Nov 03 18:07:14.148175 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:14.148010 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:14.228083 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:14.228047 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" event={"ID":"44286112-623b-4654-9b84-7eed4dd067d9","Type":"ContainerStarted","Data":"a0d08bd0b2d0ab1dc5283ad828636d0c9c8e079345aa0702b8234a2cb6485127"} Nov 03 18:07:14.228233 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:14.228113 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" event={"ID":"44286112-623b-4654-9b84-7eed4dd067d9","Type":"ContainerStarted","Data":"a3b18099a3eaf82b995f2e533a30c0dabddda68b79f021beab499bc3af514263"} Nov 03 18:07:14.409235 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:14.409123 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" podStartSLOduration=1.409102899 podStartE2EDuration="1.409102899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:14.403212461 +0000 UTC m=+1271.536841473" watchObservedRunningTime="2026-11-03 18:07:14.409102899 +0000 UTC m=+1271.542731901" Nov 03 18:07:14.539664 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:14.539359 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:07:14.539664 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:14.539466 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:07:14.539664 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:14.539527 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:07:14.544888 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:14.544852 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw"] Nov 03 18:07:14.587923 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:14.587892 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7"] Nov 03 18:07:14.603800 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:14.603757 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:14.628648 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:14.627165 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:07:14.628648 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:14.627257 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:07:14.628648 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:14.627302 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:07:14.642028 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:14.641987 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:15.060514 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:15.048601 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:15.243549 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:15.242346 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" event={"ID":"7a819308-670a-4a83-8512-630337c61d5a","Type":"ContainerStarted","Data":"c6b9756f2536b09d7a58f7c3b85de440249c2f523e8e2bcab3a53fed2f27e604"} Nov 03 18:07:15.243549 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:15.242441 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" event={"ID":"7a819308-670a-4a83-8512-630337c61d5a","Type":"ContainerStarted","Data":"fd84daf29a2e782b2a27368bbf627d96d99259e84e57a443a1730007e958c0ab"} Nov 03 18:07:15.248296 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:15.248256 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" event={"ID":"7d997ac9-3961-424f-88c5-7b2c2168eab9","Type":"ContainerStarted","Data":"7da719c73c98f1e01ec91b213feac532001b38059a0dd10d3c91d20e1c5ea843"} Nov 03 18:07:15.249454 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:15.248535 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" event={"ID":"7d997ac9-3961-424f-88c5-7b2c2168eab9","Type":"ContainerStarted","Data":"d9cdb0f5e663b66f1632256724960d74c1213abc2d7525c1c57f7d39839f0fa1"} Nov 03 18:07:15.254035 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:15.253221 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:15.385266 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:15.385160 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" podStartSLOduration=2.385141068 podStartE2EDuration="2.385141068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:15.372166583 +0000 UTC m=+1272.505795594" watchObservedRunningTime="2026-11-03 18:07:15.385141068 +0000 UTC m=+1272.518770082" Nov 03 18:07:15.514480 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:15.514414 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" podStartSLOduration=2.5143761380000003 podStartE2EDuration="2.514376138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:15.508584698 +0000 UTC m=+1272.642213736" watchObservedRunningTime="2026-11-03 18:07:15.514376138 +0000 UTC m=+1272.648005150" Nov 03 18:07:15.771731 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:15.771647 5805 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 18:07:15.772040 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:15.772010 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" podUID="44286112-623b-4654-9b84-7eed4dd067d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 18:07:16.164961 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:16.164914 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:16.165255 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:16.165237 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:16.180547 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:16.180511 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 18:07:16.180833 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:16.180804 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" podUID="7d997ac9-3961-424f-88c5-7b2c2168eab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 18:07:16.193714 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:16.193672 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 18:07:16.194010 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:16.193976 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" podUID="7a819308-670a-4a83-8512-630337c61d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 18:07:16.509523 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:16.509481 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:16.643316 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:16.643283 5805 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 18:07:16.643990 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:16.643957 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" podUID="44286112-623b-4654-9b84-7eed4dd067d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 18:07:16.693630 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:16.693570 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:16.762714 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:16.762602 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" podUID="2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 18:07:16.825225 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:16.825171 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" podUID="e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:17.072584 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:17.072539 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" podUID="ec4a7156-46e6-4274-9794-e6d1f14aec3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:17.078509 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:17.078469 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:17.145421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:17.120498 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" podUID="277a1fec-5c82-449d-839f-e843fa0f813d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:17.145421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:17.121033 5805 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" podUID="2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Nov 03 18:07:17.145421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:17.121582 5805 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" podUID="ec4a7156-46e6-4274-9794-e6d1f14aec3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": context deadline exceeded" Nov 03 18:07:17.145421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:17.124737 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 18:07:17.145421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:17.124777 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" podUID="7a819308-670a-4a83-8512-630337c61d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 18:07:17.150416 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:17.149064 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 18:07:17.150416 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:17.149137 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" podUID="7d997ac9-3961-424f-88c5-7b2c2168eab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 18:07:17.456527 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:17.456422 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:17.582719 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:17.582678 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 03 18:07:17.601831 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:17.597385 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:17.602439 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:17.602413 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 18:07:17.732926 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:17.732825 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:17.744022 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:17.743991 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 18:07:17.829423 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:17.763493 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" podUID="2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:17.829423 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:17.789259 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:07:17.900724 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:17.900683 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" podUID="e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 03 18:07:17.905418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:17.901884 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 18:07:18.030635 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.030541 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" podUID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 03 18:07:18.091427 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.040895 5805 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" podUID="e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded" Nov 03 18:07:18.091427 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.040947 5805 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:18.135421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.121367 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" podUID="277a1fec-5c82-449d-839f-e843fa0f813d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:18.135421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.122773 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" podUID="ec4a7156-46e6-4274-9794-e6d1f14aec3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 18:07:18.135421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.123891 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 18:07:18.135421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.128513 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:07:18.135421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.128572 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:07:18.135421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.128607 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:07:18.160421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.157371 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 18:07:18.160421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.157456 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" podUID="7a819308-670a-4a83-8512-630337c61d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 18:07:18.160421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.158440 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 18:07:18.160421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.158480 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" podUID="7d997ac9-3961-424f-88c5-7b2c2168eab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 18:07:18.160421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.158551 5805 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:18.218419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.217507 5805 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 18:07:18.218419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.217572 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" podUID="44286112-623b-4654-9b84-7eed4dd067d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 18:07:18.286474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.285383 5805 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" podUID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:18.557578 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.479480 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Nov 03 18:07:18.557578 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.480852 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:07:18.645417 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.628557 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:18.670419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.648997 5805 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 18:07:18.670419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.649053 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" podUID="44286112-623b-4654-9b84-7eed4dd067d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 18:07:18.670419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.649178 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Nov 03 18:07:18.670419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.649441 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 18:07:18.810473 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.799518 5805 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:07:18.810473 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.799575 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" podUID="f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:18.810473 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.799648 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 03 18:07:18.810473 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.800347 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" podUID="2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:18.902514 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:18.902469 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" podUID="e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Nov 03 18:07:19.094509 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:19.094364 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" podUID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:19.121639 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:19.121595 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" podUID="277a1fec-5c82-449d-839f-e843fa0f813d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded" Nov 03 18:07:19.149425 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:19.133881 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 18:07:19.149425 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:19.134439 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" podUID="7a819308-670a-4a83-8512-630337c61d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 18:07:19.149425 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:19.133981 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" podUID="ec4a7156-46e6-4274-9794-e6d1f14aec3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Nov 03 18:07:19.149425 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:19.133937 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:19.152829 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:19.152797 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 18:07:19.154498 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:19.154461 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" podUID="7d997ac9-3961-424f-88c5-7b2c2168eab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 18:07:19.179303 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:19.179275 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 18:07:19.306420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:19.305228 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:07:19.493941 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:19.493051 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:19.646522 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:19.646475 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 03 18:07:19.711423 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:19.666902 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:19.729431 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:19.689993 5805 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 18:07:19.729431 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:19.729278 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" podUID="44286112-623b-4654-9b84-7eed4dd067d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 18:07:19.803471 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:19.802538 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 03 18:07:19.805588 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:19.803940 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" podUID="2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:19.840635 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:19.839184 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 18:07:19.857417 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:19.846131 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:07:20.097425 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:20.096254 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" podUID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:20.180452 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:20.179610 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:20.207420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:20.207219 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 18:07:20.207420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:20.207276 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" podUID="7d997ac9-3961-424f-88c5-7b2c2168eab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 18:07:20.211531 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:20.208117 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 18:07:20.211531 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:20.208170 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" podUID="7a819308-670a-4a83-8512-630337c61d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 18:07:20.507547 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:20.503518 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:20.526423 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:20.519673 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:07:20.714908 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:20.653187 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 03 18:07:20.719730 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:20.719322 5805 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 18:07:20.719730 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:20.719384 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" podUID="44286112-623b-4654-9b84-7eed4dd067d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 18:07:20.773124 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:20.763077 5805 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Nov 03 18:07:20.811067 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:20.811011 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:20.821259 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:20.812519 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:20.909412 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:20.908452 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" podUID="e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:21.040180 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:21.039919 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:21.102742 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:21.102681 5805 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:21.106417 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:21.103053 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" podUID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Nov 03 18:07:21.222430 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:21.158488 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 18:07:21.222430 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:21.158549 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" podUID="7d997ac9-3961-424f-88c5-7b2c2168eab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 18:07:21.222430 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:21.159119 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 18:07:21.222430 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:21.159158 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" podUID="7a819308-670a-4a83-8512-630337c61d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 18:07:21.222430 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:21.191715 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Nov 03 18:07:21.222430 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:21.209493 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" podUID="277a1fec-5c82-449d-839f-e843fa0f813d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:21.280272 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:21.280235 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:21.504142 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:21.504084 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:21.669416 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:21.668929 5805 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 18:07:21.669416 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:21.669004 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" podUID="44286112-623b-4654-9b84-7eed4dd067d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 18:07:21.739017 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:21.738979 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:21.821459 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:21.814029 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:21.846418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:21.844467 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:21.931420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:21.923011 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 18:07:21.957220 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:21.957176 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" podUID="e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:21.977419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:21.957771 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 18:07:22.016410 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:22.013687 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 18:07:22.048419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:22.048314 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 18:07:22.048860 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:22.048840 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:07:22.149084 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:22.148992 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:07:22.174620 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:22.174587 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 18:07:22.174887 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:22.174854 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" podUID="7d997ac9-3961-424f-88c5-7b2c2168eab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 18:07:22.213808 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:22.213775 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 18:07:22.215583 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:22.215534 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 18:07:22.215780 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:22.215753 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" podUID="7a819308-670a-4a83-8512-630337c61d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 18:07:22.325564 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:22.325536 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 18:07:22.645083 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:22.645047 5805 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 18:07:22.655514 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:22.655479 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" podUID="44286112-623b-4654-9b84-7eed4dd067d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 18:07:23.193582 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:23.190215 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 18:07:23.193582 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:23.190283 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" podUID="7d997ac9-3961-424f-88c5-7b2c2168eab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 18:07:23.282609 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:23.282571 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 18:07:23.282880 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:23.282845 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" podUID="7a819308-670a-4a83-8512-630337c61d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 18:07:23.703026 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:23.702988 5805 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 18:07:23.704663 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:23.704496 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" podUID="44286112-623b-4654-9b84-7eed4dd067d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 18:07:24.178230 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:24.178194 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 18:07:24.178538 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:24.178508 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" podUID="7d997ac9-3961-424f-88c5-7b2c2168eab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 18:07:24.189417 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:24.179021 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 18:07:24.189660 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:24.189630 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" podUID="7a819308-670a-4a83-8512-630337c61d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 18:07:24.642151 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:24.642112 5805 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 03 18:07:24.642703 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:24.642171 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" podUID="44286112-623b-4654-9b84-7eed4dd067d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 03 18:07:25.126792 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:25.126757 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 03 18:07:25.127560 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:25.127500 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" podUID="7a819308-670a-4a83-8512-630337c61d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 03 18:07:25.153539 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:25.153509 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 18:07:25.153963 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:25.153935 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" podUID="7d997ac9-3961-424f-88c5-7b2c2168eab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 18:07:26.095623 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:26.095536 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:26.214353 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:26.214311 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:26.215812 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:26.215792 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:26.224385 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:26.224362 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:07:26.253017 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:07:26.252979 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:07:26.325423 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:26.324693 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:26.326064 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:26.326041 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:26.327331 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:26.327303 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:07:26.683582 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:26.683537 5805 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:07:26.683758 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:26.683615 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" podUID="44286112-623b-4654-9b84-7eed4dd067d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:26.990371 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:26.990276 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:26.990691 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:26.990665 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:26.993867 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:26.993770 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:07:28.436745 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.436712 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm"] Nov 03 18:07:28.441114 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.441093 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.507349 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.507304 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.507533 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.507409 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmp9c\" (UniqueName: \"kubernetes.io/projected/348445a7-ec7a-454d-86aa-8ef0beeccb33-kube-api-access-pmp9c\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.507533 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.507507 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.507668 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.507591 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.507734 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.507666 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.507734 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.507702 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.507835 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.507732 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/348445a7-ec7a-454d-86aa-8ef0beeccb33-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.507835 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.507824 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.512246 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.507892 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.537860 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.537786 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm"] Nov 03 18:07:28.572046 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.572008 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:28.609498 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.609470 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.609654 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.609524 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmp9c\" (UniqueName: \"kubernetes.io/projected/348445a7-ec7a-454d-86aa-8ef0beeccb33-kube-api-access-pmp9c\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.609654 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.609603 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.609741 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.609677 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.609779 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.609743 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.609779 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.609771 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.609854 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.609799 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/348445a7-ec7a-454d-86aa-8ef0beeccb33-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.609901 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.609879 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.609955 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.609926 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.610117 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.610092 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-credential-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.610204 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.610185 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-data\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.610410 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.610373 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-workload-socket\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.610410 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.610383 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-workload-certs\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.610711 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.610689 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/348445a7-ec7a-454d-86aa-8ef0beeccb33-istiod-ca-cert\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.612318 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.612296 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-podinfo\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.612435 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.612317 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-envoy\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.676410 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.676346 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-token\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.696256 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.696176 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmp9c\" (UniqueName: \"kubernetes.io/projected/348445a7-ec7a-454d-86aa-8ef0beeccb33-kube-api-access-pmp9c\") pod \"e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.757447 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.757147 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:28.790514 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:28.790466 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:29.218546 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:29.218482 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm"] Nov 03 18:07:29.223422 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:07:29.223370 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348445a7_ec7a_454d_86aa_8ef0beeccb33.slice/crio-3c1b5debeb9e61891c11269908c1edad4c0a02b6d43a549be506f142fe7aa7e4 WatchSource:0}: Error finding container 3c1b5debeb9e61891c11269908c1edad4c0a02b6d43a549be506f142fe7aa7e4: Status 404 returned error can't find the container with id 3c1b5debeb9e61891c11269908c1edad4c0a02b6d43a549be506f142fe7aa7e4 Nov 03 18:07:29.226316 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:29.226281 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:07:29.226457 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:29.226353 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:07:29.226457 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:29.226387 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:07:29.594013 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:29.593723 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" event={"ID":"348445a7-ec7a-454d-86aa-8ef0beeccb33","Type":"ContainerStarted","Data":"265c19c926adecfebfcbc5c7d777a8aab146a3624b3db170f0915a6532746351"} Nov 03 18:07:29.594527 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:29.594049 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" event={"ID":"348445a7-ec7a-454d-86aa-8ef0beeccb33","Type":"ContainerStarted","Data":"3c1b5debeb9e61891c11269908c1edad4c0a02b6d43a549be506f142fe7aa7e4"} Nov 03 18:07:29.726191 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:29.726089 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" podStartSLOduration=1.726066714 podStartE2EDuration="1.726066714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:29.708558198 +0000 UTC m=+1286.842187226" watchObservedRunningTime="2026-11-03 18:07:29.726066714 +0000 UTC m=+1286.859695728" Nov 03 18:07:29.757920 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:29.757886 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:30.270088 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:30.270054 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:30.483432 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:30.483383 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:30.706527 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:30.706492 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:30.759275 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:30.759225 5805 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 18:07:30.759547 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:30.759296 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" podUID="348445a7-ec7a-454d-86aa-8ef0beeccb33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 18:07:30.943422 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:30.930027 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:31.145420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:31.144336 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:31.757981 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:31.757951 5805 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 18:07:31.758363 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:31.758003 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" podUID="348445a7-ec7a-454d-86aa-8ef0beeccb33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 18:07:32.758184 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:32.758054 5805 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 18:07:32.758184 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:32.758151 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" podUID="348445a7-ec7a-454d-86aa-8ef0beeccb33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 18:07:33.758430 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:33.758383 5805 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 18:07:33.758831 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:33.758459 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" podUID="348445a7-ec7a-454d-86aa-8ef0beeccb33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 18:07:34.758370 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:34.758322 5805 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 18:07:34.758541 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:34.758425 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" podUID="348445a7-ec7a-454d-86aa-8ef0beeccb33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 18:07:35.516470 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:35.516435 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:35.719558 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:35.719509 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:35.758577 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:35.758545 5805 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Nov 03 18:07:35.758943 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:35.758604 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" podUID="348445a7-ec7a-454d-86aa-8ef0beeccb33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Nov 03 18:07:37.110140 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:37.109730 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:37.111673 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:37.111650 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:37.113807 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:37.113787 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:07:40.270806 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:40.270776 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:42.096693 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.096653 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:42.570656 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.570617 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd"] Nov 03 18:07:42.579703 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.579680 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.667648 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.667616 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd"] Nov 03 18:07:42.684588 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.684560 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.684759 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.684602 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.684759 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.684644 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.684857 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.684774 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq9mw\" (UniqueName: \"kubernetes.io/projected/51204cd4-e74e-4d85-acf7-2118f81797d2-kube-api-access-mq9mw\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.684894 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.684879 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.684980 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.684957 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51204cd4-e74e-4d85-acf7-2118f81797d2-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.685085 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.685069 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.685229 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.685215 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.685286 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.685270 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.786943 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.786907 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.787131 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.786980 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.787131 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.787024 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.787131 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.787061 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.787131 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.787092 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.787131 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.787121 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq9mw\" (UniqueName: \"kubernetes.io/projected/51204cd4-e74e-4d85-acf7-2118f81797d2-kube-api-access-mq9mw\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.787341 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.787168 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.787341 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.787192 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51204cd4-e74e-4d85-acf7-2118f81797d2-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.787341 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.787262 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.787890 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.787861 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-data\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.787890 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.787877 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-credential-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.788046 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.787954 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-workload-certs\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.788213 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.788199 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-workload-socket\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.788302 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.788279 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51204cd4-e74e-4d85-acf7-2118f81797d2-istiod-ca-cert\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.790153 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.790103 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-podinfo\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.790261 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.790221 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-envoy\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.847355 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.847252 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-token\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.847545 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.847450 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq9mw\" (UniqueName: \"kubernetes.io/projected/51204cd4-e74e-4d85-acf7-2118f81797d2-kube-api-access-mq9mw\") pod \"e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:42.895060 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:42.895022 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:43.328512 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:43.328480 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd"] Nov 03 18:07:43.331324 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:07:43.331287 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51204cd4_e74e_4d85_acf7_2118f81797d2.slice/crio-b4cea092ddf8cddaa811459da523b730a2a8f7372be42d0b44c2c7679c347d60 WatchSource:0}: Error finding container b4cea092ddf8cddaa811459da523b730a2a8f7372be42d0b44c2c7679c347d60: Status 404 returned error can't find the container with id b4cea092ddf8cddaa811459da523b730a2a8f7372be42d0b44c2c7679c347d60 Nov 03 18:07:43.333338 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:43.333303 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:07:43.333439 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:43.333369 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:07:43.333439 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:43.333413 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:07:43.746816 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:43.746734 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" event={"ID":"51204cd4-e74e-4d85-acf7-2118f81797d2","Type":"ContainerStarted","Data":"af6d367ce78fed85fc8d085cbb53215c8a3acfd13e1caf94be1885193cff1ec9"} Nov 03 18:07:43.746816 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:43.746784 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" event={"ID":"51204cd4-e74e-4d85-acf7-2118f81797d2","Type":"ContainerStarted","Data":"b4cea092ddf8cddaa811459da523b730a2a8f7372be42d0b44c2c7679c347d60"} Nov 03 18:07:43.865423 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:43.862761 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" podStartSLOduration=1.8627406359999998 podStartE2EDuration="1.862740636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:43.861235151 +0000 UTC m=+1300.994864181" watchObservedRunningTime="2026-11-03 18:07:43.862740636 +0000 UTC m=+1300.996369648" Nov 03 18:07:43.895943 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:43.895911 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:44.897109 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:44.897075 5805 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 18:07:44.897520 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:44.897128 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" podUID="51204cd4-e74e-4d85-acf7-2118f81797d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 18:07:45.062418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:45.061220 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:45.895422 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:45.895367 5805 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 18:07:45.895678 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:45.895451 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" podUID="51204cd4-e74e-4d85-acf7-2118f81797d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 18:07:46.896389 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:46.896353 5805 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 18:07:46.896827 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:46.896440 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" podUID="51204cd4-e74e-4d85-acf7-2118f81797d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 18:07:47.895770 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:47.895733 5805 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 18:07:47.895964 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:47.895797 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" podUID="51204cd4-e74e-4d85-acf7-2118f81797d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 18:07:48.896405 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:48.896352 5805 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 18:07:48.896789 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:48.896451 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" podUID="51204cd4-e74e-4d85-acf7-2118f81797d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 18:07:49.660596 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:49.660562 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:49.895700 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:49.895661 5805 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 18:07:49.895912 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:49.895731 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" podUID="51204cd4-e74e-4d85-acf7-2118f81797d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 18:07:50.900105 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:50.900074 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:50.900570 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:50.900552 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:50.901238 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:50.901216 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:07:52.870182 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:52.870142 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:53.090407 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:53.090355 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:54.158850 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:54.158814 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:54.393359 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:54.393316 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:07:56.237863 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:07:56.237821 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:07:58.823647 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:07:58.823608 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:08:03.166221 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:03.166187 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84"] Nov 03 18:08:03.171309 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:03.171284 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" Nov 03 18:08:03.223736 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:03.223707 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4b6e7259-serving-cert\"" Nov 03 18:08:03.235847 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:03.235808 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqdh2\" (UniqueName: \"kubernetes.io/projected/d0962242-28ca-4928-8594-bd4220393acc-kube-api-access-dqdh2\") pod \"maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84\" (UID: \"d0962242-28ca-4928-8594-bd4220393acc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" Nov 03 18:08:03.236275 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:03.236242 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0962242-28ca-4928-8594-bd4220393acc-metrics-certs\") pod \"maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84\" (UID: \"d0962242-28ca-4928-8594-bd4220393acc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" Nov 03 18:08:03.236684 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:03.236653 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0962242-28ca-4928-8594-bd4220393acc-maas-api-tls\") pod \"maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84\" (UID: \"d0962242-28ca-4928-8594-bd4220393acc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" Nov 03 18:08:03.236813 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:03.236757 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0962242-28ca-4928-8594-bd4220393acc-openshift-service-ca\") pod \"maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84\" (UID: \"d0962242-28ca-4928-8594-bd4220393acc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" Nov 03 18:08:03.275965 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:03.275924 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84"] Nov 03 18:08:03.337823 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:03.337789 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0962242-28ca-4928-8594-bd4220393acc-metrics-certs\") pod \"maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84\" (UID: \"d0962242-28ca-4928-8594-bd4220393acc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" Nov 03 18:08:03.337982 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:03.337843 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0962242-28ca-4928-8594-bd4220393acc-maas-api-tls\") pod \"maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84\" (UID: \"d0962242-28ca-4928-8594-bd4220393acc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" Nov 03 18:08:03.337982 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:03.337920 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0962242-28ca-4928-8594-bd4220393acc-openshift-service-ca\") pod \"maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84\" (UID: \"d0962242-28ca-4928-8594-bd4220393acc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" Nov 03 18:08:03.338104 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:08:03.338024 5805 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-serving-cert: secret "maas-api-e2e-ait-4b6e7259-serving-cert" not found Nov 03 18:08:03.338104 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:03.338095 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqdh2\" (UniqueName: \"kubernetes.io/projected/d0962242-28ca-4928-8594-bd4220393acc-kube-api-access-dqdh2\") pod \"maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84\" (UID: \"d0962242-28ca-4928-8594-bd4220393acc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" Nov 03 18:08:03.338444 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:08:03.338421 5805 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0962242-28ca-4928-8594-bd4220393acc-maas-api-tls podName:d0962242-28ca-4928-8594-bd4220393acc nodeName:}" failed. No retries permitted until 2026-11-03 18:08:03.838385451 +0000 UTC m=+1320.972014448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d0962242-28ca-4928-8594-bd4220393acc-maas-api-tls") pod "maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" (UID: "d0962242-28ca-4928-8594-bd4220393acc") : secret "maas-api-e2e-ait-4b6e7259-serving-cert" not found Nov 03 18:08:03.338764 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:03.338737 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0962242-28ca-4928-8594-bd4220393acc-openshift-service-ca\") pod \"maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84\" (UID: \"d0962242-28ca-4928-8594-bd4220393acc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" Nov 03 18:08:03.340558 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:03.340533 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0962242-28ca-4928-8594-bd4220393acc-metrics-certs\") pod \"maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84\" (UID: \"d0962242-28ca-4928-8594-bd4220393acc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" Nov 03 18:08:03.404699 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:03.404623 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqdh2\" (UniqueName: \"kubernetes.io/projected/d0962242-28ca-4928-8594-bd4220393acc-kube-api-access-dqdh2\") pod \"maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84\" (UID: \"d0962242-28ca-4928-8594-bd4220393acc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" Nov 03 18:08:03.423750 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:03.423653 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:08:03.638917 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:03.638887 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:08:03.845908 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:03.845877 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0962242-28ca-4928-8594-bd4220393acc-maas-api-tls\") pod \"maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84\" (UID: \"d0962242-28ca-4928-8594-bd4220393acc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" Nov 03 18:08:03.848544 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:03.848520 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0962242-28ca-4928-8594-bd4220393acc-maas-api-tls\") pod \"maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84\" (UID: \"d0962242-28ca-4928-8594-bd4220393acc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" Nov 03 18:08:03.849878 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:03.849856 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:08:04.087198 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:04.087164 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" Nov 03 18:08:04.467524 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:04.467425 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84"] Nov 03 18:08:04.471924 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:08:04.471896 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0962242_28ca_4928_8594_bd4220393acc.slice/crio-cdc10236b88b77494a7b745f66eecf43a67e84ce6cb5a9c81a3b6606c8b5509a WatchSource:0}: Error finding container cdc10236b88b77494a7b745f66eecf43a67e84ce6cb5a9c81a3b6606c8b5509a: Status 404 returned error can't find the container with id cdc10236b88b77494a7b745f66eecf43a67e84ce6cb5a9c81a3b6606c8b5509a Nov 03 18:08:04.953745 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:04.953710 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" event={"ID":"d0962242-28ca-4928-8594-bd4220393acc","Type":"ContainerStarted","Data":"cdc10236b88b77494a7b745f66eecf43a67e84ce6cb5a9c81a3b6606c8b5509a"} Nov 03 18:08:05.962624 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:05.962589 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" event={"ID":"d0962242-28ca-4928-8594-bd4220393acc","Type":"ContainerStarted","Data":"62cce894c859823ca3a3a6998ba4d5dd97c92925f7550b559c01fcc58db9c90c"} Nov 03 18:08:05.963116 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:05.962675 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" Nov 03 18:08:06.082107 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:06.082050 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" podStartSLOduration=2.16022574 podStartE2EDuration="3.082035407s" totalImagesPullingTime="921.809667ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:08:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:08:06.077537225 +0000 UTC m=+1323.211166260" watchObservedRunningTime="2026-11-03 18:08:06.082035407 +0000 UTC m=+1323.215664442" Nov 03 18:08:08.077331 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:08.077291 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:08:10.700466 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.700431 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc"] Nov 03 18:08:10.705219 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.705193 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.813386 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.813337 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc"] Nov 03 18:08:10.846581 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.846551 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-29bb4e29-openshift-default-dockercfg-c7vc8\"" Nov 03 18:08:10.850360 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.850329 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/836e70f4-67c5-4fec-8e50-218aae378380-istio-token\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.850531 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.850512 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836e70f4-67c5-4fec-8e50-218aae378380-istiod-ca-cert\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.850574 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.850552 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-istio-data\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.850612 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.850586 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-istio-envoy\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.850763 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.850743 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcvlj\" (UniqueName: \"kubernetes.io/projected/836e70f4-67c5-4fec-8e50-218aae378380-kube-api-access-xcvlj\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.850826 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.850783 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836e70f4-67c5-4fec-8e50-218aae378380-istio-podinfo\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.850862 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.850847 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-workload-socket\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.850898 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.850871 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-workload-certs\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.850951 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.850936 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-credential-socket\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.952937 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.952849 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836e70f4-67c5-4fec-8e50-218aae378380-istiod-ca-cert\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.952937 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.952896 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-istio-data\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.952937 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.952926 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-istio-envoy\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.953197 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.953047 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcvlj\" (UniqueName: \"kubernetes.io/projected/836e70f4-67c5-4fec-8e50-218aae378380-kube-api-access-xcvlj\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.953197 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.953084 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836e70f4-67c5-4fec-8e50-218aae378380-istio-podinfo\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.953197 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.953137 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-workload-socket\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.953349 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.953253 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-workload-certs\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.953441 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.953357 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-credential-socket\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.953519 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.953478 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-workload-socket\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.953638 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.953356 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-istio-data\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.953638 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.953613 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-workload-certs\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.953756 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.953673 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836e70f4-67c5-4fec-8e50-218aae378380-istiod-ca-cert\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.953756 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.953699 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-credential-socket\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.953917 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.953796 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/836e70f4-67c5-4fec-8e50-218aae378380-istio-token\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.955572 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.955551 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-istio-envoy\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:10.955677 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:10.955633 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836e70f4-67c5-4fec-8e50-218aae378380-istio-podinfo\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:11.010920 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:11.010891 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/836e70f4-67c5-4fec-8e50-218aae378380-istio-token\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:11.011068 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:11.010978 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcvlj\" (UniqueName: \"kubernetes.io/projected/836e70f4-67c5-4fec-8e50-218aae378380-kube-api-access-xcvlj\") pod \"e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:11.020080 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:11.020048 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:11.459249 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:11.459220 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc"] Nov 03 18:08:11.462752 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:08:11.462726 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod836e70f4_67c5_4fec_8e50_218aae378380.slice/crio-6cfc235828f2bd2c7e1637b7860bb82ce3d08d3caad1f763abac233b680c4cdf WatchSource:0}: Error finding container 6cfc235828f2bd2c7e1637b7860bb82ce3d08d3caad1f763abac233b680c4cdf: Status 404 returned error can't find the container with id 6cfc235828f2bd2c7e1637b7860bb82ce3d08d3caad1f763abac233b680c4cdf Nov 03 18:08:11.465244 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:11.465211 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:08:11.465338 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:11.465279 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:08:11.465378 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:11.465336 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:08:11.977414 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:11.976932 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" Nov 03 18:08:12.005238 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:12.005205 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" event={"ID":"836e70f4-67c5-4fec-8e50-218aae378380","Type":"ContainerStarted","Data":"109dd43128e387c3d1d2b2a75e10bd911c0e7ade722bd97b82404c2213cf6365"} Nov 03 18:08:12.005433 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:12.005263 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" event={"ID":"836e70f4-67c5-4fec-8e50-218aae378380","Type":"ContainerStarted","Data":"6cfc235828f2bd2c7e1637b7860bb82ce3d08d3caad1f763abac233b680c4cdf"} Nov 03 18:08:12.026093 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:12.026052 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:12.028001 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:12.027960 5805 patch_prober.go:29] interesting pod/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 18:08:12.028170 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:12.028036 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" podUID="836e70f4-67c5-4fec-8e50-218aae378380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 18:08:12.238595 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:12.238379 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" podStartSLOduration=2.238360252 podStartE2EDuration="2.238360252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:08:12.220449809 +0000 UTC m=+1329.354078813" watchObservedRunningTime="2026-11-03 18:08:12.238360252 +0000 UTC m=+1329.371989261" Nov 03 18:08:12.588274 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:12.588238 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:08:12.801227 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:12.801196 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:08:13.045481 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:13.042509 5805 patch_prober.go:29] interesting pod/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 18:08:13.045481 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:13.042575 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" podUID="836e70f4-67c5-4fec-8e50-218aae378380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 18:08:14.026419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:14.024510 5805 patch_prober.go:29] interesting pod/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 18:08:14.026419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:14.024581 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" podUID="836e70f4-67c5-4fec-8e50-218aae378380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 18:08:15.047419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:15.045807 5805 patch_prober.go:29] interesting pod/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 18:08:15.047419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:15.045881 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" podUID="836e70f4-67c5-4fec-8e50-218aae378380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 18:08:16.040422 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:16.033751 5805 patch_prober.go:29] interesting pod/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 18:08:16.040422 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:16.033823 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" podUID="836e70f4-67c5-4fec-8e50-218aae378380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 18:08:16.674247 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:16.674215 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:08:17.044479 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:17.043802 5805 patch_prober.go:29] interesting pod/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 03 18:08:17.044479 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:17.043869 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" podUID="836e70f4-67c5-4fec-8e50-218aae378380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 03 18:08:17.269999 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:17.266500 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:08:18.025793 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:18.025762 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:18.026182 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:18.026035 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:18.026899 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:18.026881 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:08:21.255795 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:21.255764 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:08:21.483426 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:21.480316 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:08:26.284073 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:08:26.281204 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:08:26.441013 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:26.440968 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:08:27.577098 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:27.577061 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:08:31.462466 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:31.462387 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:08:33.950898 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:33.949453 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:08:34.155780 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:34.155740 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:08:36.168372 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:36.168339 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:08:41.128519 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:41.128478 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:08:45.929116 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:45.929028 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:08:48.765148 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:48.765114 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh"] Nov 03 18:08:49.109709 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:49.109043 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:08:49.305752 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:49.305709 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:08:50.716081 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:50.716043 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:08:53.765677 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:53.765637 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh" podUID="e554043b-309f-4fa8-ae38-77211123fb58" containerName="payload-pre-processing" containerID="cri-o://38d53ac9a0e78be1f1de2defe7debc487debdd97988c168df51bf011f65d9bb4" gracePeriod=25 Nov 03 18:08:54.040310 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:54.040283 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh" Nov 03 18:08:54.114190 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:54.114156 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e554043b-309f-4fa8-ae38-77211123fb58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e554043b-309f-4fa8-ae38-77211123fb58-plugins-config-volume\") pod \"e554043b-309f-4fa8-ae38-77211123fb58\" (UID: \"e554043b-309f-4fa8-ae38-77211123fb58\") " Nov 03 18:08:54.114372 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:54.114254 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e554043b-309f-4fa8-ae38-77211123fb58-kube-api-access-brwz4\" (UniqueName: \"kubernetes.io/projected/e554043b-309f-4fa8-ae38-77211123fb58-kube-api-access-brwz4\") pod \"e554043b-309f-4fa8-ae38-77211123fb58\" (UID: \"e554043b-309f-4fa8-ae38-77211123fb58\") " Nov 03 18:08:54.114603 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:54.114576 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e554043b-309f-4fa8-ae38-77211123fb58-plugins-config-volume" pod "e554043b-309f-4fa8-ae38-77211123fb58" (UID: "e554043b-309f-4fa8-ae38-77211123fb58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:08:54.115017 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:54.114979 5805 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e554043b-309f-4fa8-ae38-77211123fb58-plugins-config-volume\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:08:54.116525 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:54.116497 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e554043b-309f-4fa8-ae38-77211123fb58-kube-api-access-brwz4" pod "e554043b-309f-4fa8-ae38-77211123fb58" (UID: "e554043b-309f-4fa8-ae38-77211123fb58"). InnerVolumeSpecName "kube-api-access-brwz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:08:54.216970 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:54.216938 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brwz4\" (UniqueName: \"kubernetes.io/projected/e554043b-309f-4fa8-ae38-77211123fb58-kube-api-access-brwz4\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:08:54.362485 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:54.362450 5805 generic.go:356] "Generic (PLEG): container finished" podID="e554043b-309f-4fa8-ae38-77211123fb58" containerID="38d53ac9a0e78be1f1de2defe7debc487debdd97988c168df51bf011f65d9bb4" exitCode=0 Nov 03 18:08:54.362643 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:54.362494 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh" Nov 03 18:08:54.362643 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:54.362533 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh" event={"ID":"e554043b-309f-4fa8-ae38-77211123fb58","Type":"ContainerDied","Data":"38d53ac9a0e78be1f1de2defe7debc487debdd97988c168df51bf011f65d9bb4"} Nov 03 18:08:54.362643 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:54.362587 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh" event={"ID":"e554043b-309f-4fa8-ae38-77211123fb58","Type":"ContainerDied","Data":"833261d87b30426de444de2efb053e855a40338a25ca16649021e46f3eda38cf"} Nov 03 18:08:54.362643 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:54.362615 5805 scope.go:122] "RemoveContainer" containerID="38d53ac9a0e78be1f1de2defe7debc487debdd97988c168df51bf011f65d9bb4" Nov 03 18:08:54.373053 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:54.373036 5805 scope.go:122] "RemoveContainer" containerID="38d53ac9a0e78be1f1de2defe7debc487debdd97988c168df51bf011f65d9bb4" Nov 03 18:08:54.373338 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:08:54.373317 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38d53ac9a0e78be1f1de2defe7debc487debdd97988c168df51bf011f65d9bb4\": container with ID starting with 38d53ac9a0e78be1f1de2defe7debc487debdd97988c168df51bf011f65d9bb4 not found: ID does not exist" containerID="38d53ac9a0e78be1f1de2defe7debc487debdd97988c168df51bf011f65d9bb4" Nov 03 18:08:54.373385 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:54.373346 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d53ac9a0e78be1f1de2defe7debc487debdd97988c168df51bf011f65d9bb4"} err="failed to get container status \"38d53ac9a0e78be1f1de2defe7debc487debdd97988c168df51bf011f65d9bb4\": rpc error: code = NotFound desc = could not find container \"38d53ac9a0e78be1f1de2defe7debc487debdd97988c168df51bf011f65d9bb4\": container with ID starting with 38d53ac9a0e78be1f1de2defe7debc487debdd97988c168df51bf011f65d9bb4 not found: ID does not exist" Nov 03 18:08:54.546540 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:54.546455 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh"] Nov 03 18:08:54.562641 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:54.562603 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-29bb4e29-bfff9d55f-fxkkh"] Nov 03 18:08:54.977803 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:54.977772 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e554043b-309f-4fa8-ae38-77211123fb58" path="/var/lib/kubelet/pods/e554043b-309f-4fa8-ae38-77211123fb58/volumes" Nov 03 18:08:55.287700 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.287614 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:08:55.356958 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.356925 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz"] Nov 03 18:08:55.357895 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.357868 5805 cpu_manager.go:414] "containerMap: removing container" podUID="e554043b-309f-4fa8-ae38-77211123fb58" containerName="payload-pre-processing" Nov 03 18:08:55.357895 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.357892 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e554043b-309f-4fa8-ae38-77211123fb58" containerName="payload-pre-processing" Nov 03 18:08:55.358095 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.358047 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="e554043b-309f-4fa8-ae38-77211123fb58" containerName="payload-pre-processing" Nov 03 18:08:55.363037 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.363016 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.451437 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.451388 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.451624 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.451523 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.451624 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.451618 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.451770 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.451752 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.451839 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.451824 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.451892 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.451857 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.451943 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.451899 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.451998 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.451986 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.452054 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.452017 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr86p\" (UniqueName: \"kubernetes.io/projected/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-kube-api-access-tr86p\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.456143 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.456119 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz"] Nov 03 18:08:55.553750 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.553677 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.553750 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.553735 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.553750 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.553757 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.554046 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.553785 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.554046 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.553842 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.554046 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.553857 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr86p\" (UniqueName: \"kubernetes.io/projected/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-kube-api-access-tr86p\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.554046 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.553917 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.554046 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.553969 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.554046 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.554012 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.554378 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.554145 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.554378 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.554205 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.554378 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.554246 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.554378 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.554279 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.554727 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.554702 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.556299 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.556278 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.556559 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.556539 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.611031 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.610999 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr86p\" (UniqueName: \"kubernetes.io/projected/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-kube-api-access-tr86p\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.611153 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.611011 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6669c77-7d2c-48ba-829f-85ce6e6f9f0c-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-6stkz\" (UID: \"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:55.678018 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:55.677989 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:56.107337 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:56.107305 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz"] Nov 03 18:08:56.112271 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:08:56.112245 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6669c77_7d2c_48ba_829f_85ce6e6f9f0c.slice/crio-7ae3a3e9f162dcc761a04ebc1d607070c68dc53f6bd9c7c6292abff66e0b824c WatchSource:0}: Error finding container 7ae3a3e9f162dcc761a04ebc1d607070c68dc53f6bd9c7c6292abff66e0b824c: Status 404 returned error can't find the container with id 7ae3a3e9f162dcc761a04ebc1d607070c68dc53f6bd9c7c6292abff66e0b824c Nov 03 18:08:56.114746 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:56.114714 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:08:56.114824 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:56.114803 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:08:56.114863 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:56.114831 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:08:56.237945 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:08:56.237910 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:08:56.379957 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:56.379869 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" event={"ID":"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c","Type":"ContainerStarted","Data":"21390b213284faaa2d251813534e50a5e4d8f3775089a13c7db1c6a3aff63ccf"} Nov 03 18:08:56.380134 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:56.379962 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" event={"ID":"c6669c77-7d2c-48ba-829f-85ce6e6f9f0c","Type":"ContainerStarted","Data":"7ae3a3e9f162dcc761a04ebc1d607070c68dc53f6bd9c7c6292abff66e0b824c"} Nov 03 18:08:56.495803 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:56.495738 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" podStartSLOduration=1.495724459 podStartE2EDuration="1.495724459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:08:56.492493346 +0000 UTC m=+1373.626122375" watchObservedRunningTime="2026-11-03 18:08:56.495724459 +0000 UTC m=+1373.629353468" Nov 03 18:08:56.678206 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:56.678122 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:08:57.680615 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:57.680576 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6stkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 18:08:57.681029 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:57.680653 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" podUID="c6669c77-7d2c-48ba-829f-85ce6e6f9f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 18:08:58.678355 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:58.678320 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6stkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 18:08:58.678554 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:58.678378 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" podUID="c6669c77-7d2c-48ba-829f-85ce6e6f9f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 18:08:59.678593 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:59.678559 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6stkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 18:08:59.679004 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:08:59.678613 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" podUID="c6669c77-7d2c-48ba-829f-85ce6e6f9f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 18:09:00.254387 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:00.254339 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:00.678735 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:00.678700 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6stkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 18:09:00.679136 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:00.678753 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" podUID="c6669c77-7d2c-48ba-829f-85ce6e6f9f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 18:09:01.678470 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:01.678435 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6stkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 18:09:01.678647 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:01.678489 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" podUID="c6669c77-7d2c-48ba-829f-85ce6e6f9f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 18:09:02.678553 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:02.678512 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6stkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 18:09:02.678941 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:02.678577 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" podUID="c6669c77-7d2c-48ba-829f-85ce6e6f9f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 18:09:03.678888 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:03.678852 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6stkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 18:09:03.679377 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:03.678914 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" podUID="c6669c77-7d2c-48ba-829f-85ce6e6f9f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 18:09:05.225045 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:05.223290 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:09:05.225045 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:05.224168 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:09:05.225045 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:05.224881 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" Nov 03 18:09:05.426618 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:05.426574 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:05.862102 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:05.862065 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:06.074199 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:06.074167 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:06.335025 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:06.334988 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:10.326051 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:10.326007 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:13.007884 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:13.007844 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:13.248421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:13.248356 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:13.470633 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:13.470603 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:14.006289 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:14.006260 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:14.206521 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:14.206486 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:14.425699 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:14.425660 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:14.670418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:14.661181 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:15.884507 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:15.884476 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:16.190218 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:16.187981 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:16.415222 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:16.415186 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:16.767782 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:16.767617 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:17.358728 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:17.358692 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:18.137364 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:18.134974 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:18.553456 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:18.553338 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:20.236784 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:20.236748 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:20.440997 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:20.440957 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:20.652905 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:20.652868 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:21.650869 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:21.650828 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:21.889586 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:21.889544 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:22.396257 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:22.396197 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:23.600867 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:23.600831 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc"] Nov 03 18:09:23.601329 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:23.601049 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" podUID="836e70f4-67c5-4fec-8e50-218aae378380" containerName="istio-proxy" containerID="cri-o://109dd43128e387c3d1d2b2a75e10bd911c0e7ade722bd97b82404c2213cf6365" gracePeriod=30 Nov 03 18:09:24.617954 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:24.617914 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:24.826892 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:24.826813 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:25.701051 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:25.701017 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:25.920659 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:25.920620 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:26.325465 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:09:26.325418 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:09:27.454646 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:27.454590 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:28.718131 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.718104 5805 generic.go:356] "Generic (PLEG): container finished" podID="836e70f4-67c5-4fec-8e50-218aae378380" containerID="109dd43128e387c3d1d2b2a75e10bd911c0e7ade722bd97b82404c2213cf6365" exitCode=0 Nov 03 18:09:28.718539 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.718170 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" event={"ID":"836e70f4-67c5-4fec-8e50-218aae378380","Type":"ContainerDied","Data":"109dd43128e387c3d1d2b2a75e10bd911c0e7ade722bd97b82404c2213cf6365"} Nov 03 18:09:28.877769 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.877709 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:09:28.888205 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.888182 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-istio-envoy\") pod \"836e70f4-67c5-4fec-8e50-218aae378380\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " Nov 03 18:09:28.888320 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.888246 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/836e70f4-67c5-4fec-8e50-218aae378380-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836e70f4-67c5-4fec-8e50-218aae378380-istiod-ca-cert\") pod \"836e70f4-67c5-4fec-8e50-218aae378380\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " Nov 03 18:09:28.888320 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.888283 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-workload-socket\") pod \"836e70f4-67c5-4fec-8e50-218aae378380\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " Nov 03 18:09:28.888430 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.888415 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/836e70f4-67c5-4fec-8e50-218aae378380-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836e70f4-67c5-4fec-8e50-218aae378380-istio-podinfo\") pod \"836e70f4-67c5-4fec-8e50-218aae378380\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " Nov 03 18:09:28.888472 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.888446 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-istio-data\") pod \"836e70f4-67c5-4fec-8e50-218aae378380\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " Nov 03 18:09:28.888627 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.888472 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/836e70f4-67c5-4fec-8e50-218aae378380-istio-token\" (UniqueName: \"kubernetes.io/projected/836e70f4-67c5-4fec-8e50-218aae378380-istio-token\") pod \"836e70f4-67c5-4fec-8e50-218aae378380\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " Nov 03 18:09:28.888627 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.888584 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-credential-socket\") pod \"836e70f4-67c5-4fec-8e50-218aae378380\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " Nov 03 18:09:28.888755 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.888618 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-workload-certs\") pod \"836e70f4-67c5-4fec-8e50-218aae378380\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " Nov 03 18:09:28.888755 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.888655 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-workload-socket" pod "836e70f4-67c5-4fec-8e50-218aae378380" (UID: "836e70f4-67c5-4fec-8e50-218aae378380"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:09:28.888863 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.888799 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/836e70f4-67c5-4fec-8e50-218aae378380-kube-api-access-xcvlj\" (UniqueName: \"kubernetes.io/projected/836e70f4-67c5-4fec-8e50-218aae378380-kube-api-access-xcvlj\") pod \"836e70f4-67c5-4fec-8e50-218aae378380\" (UID: \"836e70f4-67c5-4fec-8e50-218aae378380\") " Nov 03 18:09:28.888947 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.888913 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-credential-socket" pod "836e70f4-67c5-4fec-8e50-218aae378380" (UID: "836e70f4-67c5-4fec-8e50-218aae378380"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:09:28.888947 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.888929 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-workload-certs" pod "836e70f4-67c5-4fec-8e50-218aae378380" (UID: "836e70f4-67c5-4fec-8e50-218aae378380"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:09:28.889048 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.888987 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-istio-data" pod "836e70f4-67c5-4fec-8e50-218aae378380" (UID: "836e70f4-67c5-4fec-8e50-218aae378380"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:09:28.889103 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.889076 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/836e70f4-67c5-4fec-8e50-218aae378380-istiod-ca-cert" pod "836e70f4-67c5-4fec-8e50-218aae378380" (UID: "836e70f4-67c5-4fec-8e50-218aae378380"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:09:28.889607 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.889580 5805 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836e70f4-67c5-4fec-8e50-218aae378380-istiod-ca-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:09:28.889713 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.889611 5805 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-workload-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:09:28.889713 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.889627 5805 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-istio-data\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:09:28.889713 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.889641 5805 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-credential-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:09:28.889713 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.889657 5805 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-workload-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:09:28.890549 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.890525 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-istio-envoy" pod "836e70f4-67c5-4fec-8e50-218aae378380" (UID: "836e70f4-67c5-4fec-8e50-218aae378380"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:09:28.890731 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.890707 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/836e70f4-67c5-4fec-8e50-218aae378380-istio-podinfo" pod "836e70f4-67c5-4fec-8e50-218aae378380" (UID: "836e70f4-67c5-4fec-8e50-218aae378380"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:09:28.890902 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.890884 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/836e70f4-67c5-4fec-8e50-218aae378380-istio-token" pod "836e70f4-67c5-4fec-8e50-218aae378380" (UID: "836e70f4-67c5-4fec-8e50-218aae378380"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:09:28.890951 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.890909 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/836e70f4-67c5-4fec-8e50-218aae378380-kube-api-access-xcvlj" pod "836e70f4-67c5-4fec-8e50-218aae378380" (UID: "836e70f4-67c5-4fec-8e50-218aae378380"). InnerVolumeSpecName "kube-api-access-xcvlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:09:28.991076 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.991050 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcvlj\" (UniqueName: \"kubernetes.io/projected/836e70f4-67c5-4fec-8e50-218aae378380-kube-api-access-xcvlj\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:09:28.991076 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.991078 5805 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836e70f4-67c5-4fec-8e50-218aae378380-istio-envoy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:09:28.991208 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.991088 5805 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836e70f4-67c5-4fec-8e50-218aae378380-istio-podinfo\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:09:28.991208 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:28.991096 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/836e70f4-67c5-4fec-8e50-218aae378380-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:09:29.726457 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:29.726425 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" Nov 03 18:09:29.726457 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:29.726431 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc" event={"ID":"836e70f4-67c5-4fec-8e50-218aae378380","Type":"ContainerDied","Data":"6cfc235828f2bd2c7e1637b7860bb82ce3d08d3caad1f763abac233b680c4cdf"} Nov 03 18:09:29.727016 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:29.726507 5805 scope.go:122] "RemoveContainer" containerID="109dd43128e387c3d1d2b2a75e10bd911c0e7ade722bd97b82404c2213cf6365" Nov 03 18:09:29.952271 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:29.952234 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc"] Nov 03 18:09:29.997968 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:29.997881 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-29bb4e29-openshift-default-7c88d587b7-6tdqc"] Nov 03 18:09:30.897178 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:30.897139 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:30.986774 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:30.986735 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="836e70f4-67c5-4fec-8e50-218aae378380" path="/var/lib/kubelet/pods/836e70f4-67c5-4fec-8e50-218aae378380/volumes" Nov 03 18:09:31.101665 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:31.101627 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:32.023202 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:32.023167 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:35.515079 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:35.515041 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:36.594271 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:36.594238 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:40.475061 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.475026 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:40.748101 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.748013 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m"] Nov 03 18:09:40.748811 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.748787 5805 cpu_manager.go:414] "containerMap: removing container" podUID="836e70f4-67c5-4fec-8e50-218aae378380" containerName="istio-proxy" Nov 03 18:09:40.748811 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.748809 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="836e70f4-67c5-4fec-8e50-218aae378380" containerName="istio-proxy" Nov 03 18:09:40.749003 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.748977 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="836e70f4-67c5-4fec-8e50-218aae378380" containerName="istio-proxy" Nov 03 18:09:40.754173 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.754153 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.831418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.814465 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3369b197-476f-4861-b6c3-ac9bfa5464d1-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.831418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.814514 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dsbx\" (UniqueName: \"kubernetes.io/projected/3369b197-476f-4861-b6c3-ac9bfa5464d1-kube-api-access-8dsbx\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.831418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.814637 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3369b197-476f-4861-b6c3-ac9bfa5464d1-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.831418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.814727 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3369b197-476f-4861-b6c3-ac9bfa5464d1-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.831418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.814773 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3369b197-476f-4861-b6c3-ac9bfa5464d1-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.831418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.814808 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3369b197-476f-4861-b6c3-ac9bfa5464d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.831418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.814839 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3369b197-476f-4861-b6c3-ac9bfa5464d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.831418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.814936 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3369b197-476f-4861-b6c3-ac9bfa5464d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.831418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.815051 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3369b197-476f-4861-b6c3-ac9bfa5464d1-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.865778 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.865742 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m"] Nov 03 18:09:40.916778 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.916740 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3369b197-476f-4861-b6c3-ac9bfa5464d1-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.916972 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.916837 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3369b197-476f-4861-b6c3-ac9bfa5464d1-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.916972 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.916884 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3369b197-476f-4861-b6c3-ac9bfa5464d1-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.916972 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.916917 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3369b197-476f-4861-b6c3-ac9bfa5464d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.916972 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.916948 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3369b197-476f-4861-b6c3-ac9bfa5464d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.917189 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.917047 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3369b197-476f-4861-b6c3-ac9bfa5464d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.917189 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.917124 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3369b197-476f-4861-b6c3-ac9bfa5464d1-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.917305 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.917286 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3369b197-476f-4861-b6c3-ac9bfa5464d1-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.917361 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.917292 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3369b197-476f-4861-b6c3-ac9bfa5464d1-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.917361 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.917321 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dsbx\" (UniqueName: \"kubernetes.io/projected/3369b197-476f-4861-b6c3-ac9bfa5464d1-kube-api-access-8dsbx\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.918022 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.918000 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3369b197-476f-4861-b6c3-ac9bfa5464d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.918178 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.918019 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3369b197-476f-4861-b6c3-ac9bfa5464d1-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.918285 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.918261 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3369b197-476f-4861-b6c3-ac9bfa5464d1-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.918443 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.918426 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3369b197-476f-4861-b6c3-ac9bfa5464d1-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.921044 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.921022 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3369b197-476f-4861-b6c3-ac9bfa5464d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.923020 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.922973 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3369b197-476f-4861-b6c3-ac9bfa5464d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.974873 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.974845 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3369b197-476f-4861-b6c3-ac9bfa5464d1-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:40.979072 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:40.979040 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dsbx\" (UniqueName: \"kubernetes.io/projected/3369b197-476f-4861-b6c3-ac9bfa5464d1-kube-api-access-8dsbx\") pod \"maas-default-gateway-openshift-default-6dcf757899-jmn7m\" (UID: \"3369b197-476f-4861-b6c3-ac9bfa5464d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:41.072354 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:41.072319 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:41.522616 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:41.522513 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m"] Nov 03 18:09:41.528209 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:09:41.528181 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3369b197_476f_4861_b6c3_ac9bfa5464d1.slice/crio-5e4dbf2e147da0e83ef35c0f48ccffa4d87a71995e6f7f394b79afa46d97c83d WatchSource:0}: Error finding container 5e4dbf2e147da0e83ef35c0f48ccffa4d87a71995e6f7f394b79afa46d97c83d: Status 404 returned error can't find the container with id 5e4dbf2e147da0e83ef35c0f48ccffa4d87a71995e6f7f394b79afa46d97c83d Nov 03 18:09:41.531286 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:41.531249 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:09:41.531370 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:41.531327 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:09:41.531370 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:41.531355 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:09:41.877672 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:41.877631 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" event={"ID":"3369b197-476f-4861-b6c3-ac9bfa5464d1","Type":"ContainerStarted","Data":"782b3cc29d0d752f190c65a0b89ac2c83bd42f3b52c0c3c334ed1e77f4bc4e1f"} Nov 03 18:09:41.877841 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:41.877705 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" event={"ID":"3369b197-476f-4861-b6c3-ac9bfa5464d1","Type":"ContainerStarted","Data":"5e4dbf2e147da0e83ef35c0f48ccffa4d87a71995e6f7f394b79afa46d97c83d"} Nov 03 18:09:41.953868 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:41.953835 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:42.001881 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:42.001832 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" podStartSLOduration=2.001819412 podStartE2EDuration="2.001819412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:09:41.998767522 +0000 UTC m=+1419.132396554" watchObservedRunningTime="2026-11-03 18:09:42.001819412 +0000 UTC m=+1419.135448481" Nov 03 18:09:42.072727 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:42.072701 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:42.074089 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:42.074060 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jmn7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 18:09:42.074217 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:42.074114 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" podUID="3369b197-476f-4861-b6c3-ac9bfa5464d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 18:09:42.224464 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:42.224360 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:42.426215 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:42.426179 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:43.073609 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:43.073570 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jmn7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 18:09:43.074008 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:43.073634 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" podUID="3369b197-476f-4861-b6c3-ac9bfa5464d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 18:09:44.073140 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:44.073107 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jmn7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 18:09:44.073300 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:44.073166 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" podUID="3369b197-476f-4861-b6c3-ac9bfa5464d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 18:09:44.084697 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:44.084662 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:45.056432 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:45.056373 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:45.072785 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:45.072745 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jmn7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 18:09:45.072968 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:45.072804 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" podUID="3369b197-476f-4861-b6c3-ac9bfa5464d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 18:09:45.254438 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:45.254404 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:45.463591 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:45.463496 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:46.073020 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:46.072989 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jmn7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 18:09:46.073192 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:46.073046 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" podUID="3369b197-476f-4861-b6c3-ac9bfa5464d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 18:09:46.481812 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:46.481719 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84"] Nov 03 18:09:46.482255 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:46.481964 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" podUID="d0962242-28ca-4928-8594-bd4220393acc" containerName="maas-api" containerID="cri-o://62cce894c859823ca3a3a6998ba4d5dd97c92925f7550b559c01fcc58db9c90c" gracePeriod=30 Nov 03 18:09:46.565521 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:46.565463 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:46.811074 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:46.811043 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:47.095418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.080744 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jmn7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 18:09:47.095418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.080809 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" podUID="3369b197-476f-4861-b6c3-ac9bfa5464d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 18:09:47.394083 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.394006 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" Nov 03 18:09:47.394440 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.394416 5805 generic.go:356] "Generic (PLEG): container finished" podID="d0962242-28ca-4928-8594-bd4220393acc" containerID="62cce894c859823ca3a3a6998ba4d5dd97c92925f7550b559c01fcc58db9c90c" exitCode=0 Nov 03 18:09:47.394637 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.394558 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" event={"ID":"d0962242-28ca-4928-8594-bd4220393acc","Type":"ContainerDied","Data":"62cce894c859823ca3a3a6998ba4d5dd97c92925f7550b559c01fcc58db9c90c"} Nov 03 18:09:47.394752 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.394743 5805 scope.go:122] "RemoveContainer" containerID="62cce894c859823ca3a3a6998ba4d5dd97c92925f7550b559c01fcc58db9c90c" Nov 03 18:09:47.490792 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.490758 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0962242-28ca-4928-8594-bd4220393acc-kube-api-access-dqdh2\" (UniqueName: \"kubernetes.io/projected/d0962242-28ca-4928-8594-bd4220393acc-kube-api-access-dqdh2\") pod \"d0962242-28ca-4928-8594-bd4220393acc\" (UID: \"d0962242-28ca-4928-8594-bd4220393acc\") " Nov 03 18:09:47.491192 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.490847 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0962242-28ca-4928-8594-bd4220393acc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0962242-28ca-4928-8594-bd4220393acc-openshift-service-ca\") pod \"d0962242-28ca-4928-8594-bd4220393acc\" (UID: \"d0962242-28ca-4928-8594-bd4220393acc\") " Nov 03 18:09:47.491192 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.490876 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0962242-28ca-4928-8594-bd4220393acc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0962242-28ca-4928-8594-bd4220393acc-metrics-certs\") pod \"d0962242-28ca-4928-8594-bd4220393acc\" (UID: \"d0962242-28ca-4928-8594-bd4220393acc\") " Nov 03 18:09:47.491303 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.491188 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0962242-28ca-4928-8594-bd4220393acc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0962242-28ca-4928-8594-bd4220393acc-maas-api-tls\") pod \"d0962242-28ca-4928-8594-bd4220393acc\" (UID: \"d0962242-28ca-4928-8594-bd4220393acc\") " Nov 03 18:09:47.491303 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.491235 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0962242-28ca-4928-8594-bd4220393acc-openshift-service-ca" pod "d0962242-28ca-4928-8594-bd4220393acc" (UID: "d0962242-28ca-4928-8594-bd4220393acc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:09:47.491812 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.491785 5805 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0962242-28ca-4928-8594-bd4220393acc-openshift-service-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:09:47.492984 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.492958 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0962242-28ca-4928-8594-bd4220393acc-metrics-certs" pod "d0962242-28ca-4928-8594-bd4220393acc" (UID: "d0962242-28ca-4928-8594-bd4220393acc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:09:47.493174 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.493156 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0962242-28ca-4928-8594-bd4220393acc-kube-api-access-dqdh2" pod "d0962242-28ca-4928-8594-bd4220393acc" (UID: "d0962242-28ca-4928-8594-bd4220393acc"). InnerVolumeSpecName "kube-api-access-dqdh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:09:47.493292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.493266 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0962242-28ca-4928-8594-bd4220393acc-maas-api-tls" pod "d0962242-28ca-4928-8594-bd4220393acc" (UID: "d0962242-28ca-4928-8594-bd4220393acc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:09:47.565898 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.565863 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf"] Nov 03 18:09:47.566565 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.566550 5805 cpu_manager.go:414] "containerMap: removing container" podUID="d0962242-28ca-4928-8594-bd4220393acc" containerName="maas-api" Nov 03 18:09:47.566637 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.566566 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0962242-28ca-4928-8594-bd4220393acc" containerName="maas-api" Nov 03 18:09:47.566682 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.566673 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0962242-28ca-4928-8594-bd4220393acc" containerName="maas-api" Nov 03 18:09:47.571558 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.571534 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:47.593343 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.593319 5805 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0962242-28ca-4928-8594-bd4220393acc-maas-api-tls\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:09:47.593467 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.593346 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqdh2\" (UniqueName: \"kubernetes.io/projected/d0962242-28ca-4928-8594-bd4220393acc-kube-api-access-dqdh2\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:09:47.593467 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.593358 5805 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0962242-28ca-4928-8594-bd4220393acc-metrics-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:09:47.698942 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.698819 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e8856532-061a-4f17-b49c-5e92228b7c02-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhkvf\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:47.699206 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.699183 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8856532-061a-4f17-b49c-5e92228b7c02-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhkvf\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:47.699354 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.699335 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7dq6\" (UniqueName: \"kubernetes.io/projected/e8856532-061a-4f17-b49c-5e92228b7c02-kube-api-access-g7dq6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhkvf\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:47.699670 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.699646 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e8856532-061a-4f17-b49c-5e92228b7c02-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhkvf\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:47.699838 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.699820 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e8856532-061a-4f17-b49c-5e92228b7c02-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhkvf\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:47.699995 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.699976 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e8856532-061a-4f17-b49c-5e92228b7c02-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhkvf\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:47.716552 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.716511 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8856532-061a-4f17-b49c-5e92228b7c02-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhkvf\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:47.835318 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.835275 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf"] Nov 03 18:09:47.837273 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.837240 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e8856532-061a-4f17-b49c-5e92228b7c02-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhkvf\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:47.898607 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.898568 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8856532-061a-4f17-b49c-5e92228b7c02-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhkvf\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:47.898851 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.898831 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7dq6\" (UniqueName: \"kubernetes.io/projected/e8856532-061a-4f17-b49c-5e92228b7c02-kube-api-access-g7dq6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhkvf\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:47.899172 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.899156 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e8856532-061a-4f17-b49c-5e92228b7c02-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhkvf\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:47.899314 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.899300 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e8856532-061a-4f17-b49c-5e92228b7c02-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhkvf\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:47.899518 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.899502 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e8856532-061a-4f17-b49c-5e92228b7c02-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhkvf\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:47.899677 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.899661 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8856532-061a-4f17-b49c-5e92228b7c02-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhkvf\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:47.907904 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.903002 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e8856532-061a-4f17-b49c-5e92228b7c02-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhkvf\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:47.907904 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.905514 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e8856532-061a-4f17-b49c-5e92228b7c02-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhkvf\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:47.907904 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.907757 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8856532-061a-4f17-b49c-5e92228b7c02-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhkvf\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:47.909121 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.909096 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e8856532-061a-4f17-b49c-5e92228b7c02-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhkvf\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:47.914419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.911247 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e8856532-061a-4f17-b49c-5e92228b7c02-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhkvf\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:47.965182 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.965073 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7dq6\" (UniqueName: \"kubernetes.io/projected/e8856532-061a-4f17-b49c-5e92228b7c02-kube-api-access-g7dq6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhkvf\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:47.966217 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:47.966168 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8856532-061a-4f17-b49c-5e92228b7c02-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vhkvf\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:48.073900 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:48.073867 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jmn7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 18:09:48.074062 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:48.073918 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" podUID="3369b197-476f-4861-b6c3-ac9bfa5464d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 18:09:48.185964 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:48.185932 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:48.403018 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:48.402982 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" Nov 03 18:09:48.403205 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:48.402982 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" event={"ID":"d0962242-28ca-4928-8594-bd4220393acc","Type":"ContainerDied","Data":"cdc10236b88b77494a7b745f66eecf43a67e84ce6cb5a9c81a3b6606c8b5509a"} Nov 03 18:09:48.503037 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:48.503004 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:48.558630 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:48.558597 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84"] Nov 03 18:09:48.563049 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:48.563022 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84"] Nov 03 18:09:48.695918 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:48.695882 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf"] Nov 03 18:09:48.700765 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:09:48.700737 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8856532_061a_4f17_b49c_5e92228b7c02.slice/crio-4b117d02c97d727aa612bd3c9af2ce4d3c9b9f72a3cc7fa48801f4c23c821c41 WatchSource:0}: Error finding container 4b117d02c97d727aa612bd3c9af2ce4d3c9b9f72a3cc7fa48801f4c23c821c41: Status 404 returned error can't find the container with id 4b117d02c97d727aa612bd3c9af2ce4d3c9b9f72a3cc7fa48801f4c23c821c41 Nov 03 18:09:48.703827 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:48.703786 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:09:48.703903 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:48.703873 5805 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 18:09:48.790436 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:48.787865 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:48.980257 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:48.980180 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0962242-28ca-4928-8594-bd4220393acc" path="/var/lib/kubelet/pods/d0962242-28ca-4928-8594-bd4220393acc/volumes" Nov 03 18:09:49.008455 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:48.996883 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:49.073024 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:49.072985 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jmn7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 18:09:49.073187 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:49.073040 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" podUID="3369b197-476f-4861-b6c3-ac9bfa5464d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 18:09:49.258526 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:49.240730 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:49.415865 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:49.415826 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" event={"ID":"e8856532-061a-4f17-b49c-5e92228b7c02","Type":"ContainerStarted","Data":"be1e8177cb462673180709120879445ee22e3b07a232a5a475448a157a360f6e"} Nov 03 18:09:49.416086 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:49.415892 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" event={"ID":"e8856532-061a-4f17-b49c-5e92228b7c02","Type":"ContainerStarted","Data":"4b117d02c97d727aa612bd3c9af2ce4d3c9b9f72a3cc7fa48801f4c23c821c41"} Nov 03 18:09:49.416086 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:49.415920 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:49.418285 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:49.418258 5805 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vhkvf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:09:49.418459 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:49.418310 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" podUID="e8856532-061a-4f17-b49c-5e92228b7c02" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:09:49.536467 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:49.530991 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" podStartSLOduration=2.530974084 podStartE2EDuration="2.530974084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:09:49.52947023 +0000 UTC m=+1426.663099265" watchObservedRunningTime="2026-11-03 18:09:49.530974084 +0000 UTC m=+1426.664603095" Nov 03 18:09:50.044640 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:50.044592 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b6e7259-859cfc479c-p4r84" podUID="d0962242-28ca-4928-8594-bd4220393acc" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.85:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:09:50.073567 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:50.073533 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jmn7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Nov 03 18:09:50.073764 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:50.073606 5805 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" podUID="3369b197-476f-4861-b6c3-ac9bfa5464d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Nov 03 18:09:50.268836 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:50.268799 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:50.426526 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:50.426492 5805 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vhkvf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:09:50.426685 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:50.426546 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" podUID="e8856532-061a-4f17-b49c-5e92228b7c02" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:09:51.034417 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:51.029957 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:51.398419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:51.393870 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:51.614416 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:51.614029 5805 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:51.619853 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:51.616151 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:51.628167 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:51.623515 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" Nov 03 18:09:51.719295 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:51.719213 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:51.942914 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:51.942876 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:53.427156 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:53.427127 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:09:54.102897 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:54.102863 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:55.696794 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:55.696760 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:55.909830 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:55.909781 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:56.235944 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:09:56.235911 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:09:58.671237 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:58.671202 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:58.897742 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:58.897711 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:59.120406 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:59.120351 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:59.355519 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:59.355485 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:59.702718 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:59.702679 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:09:59.930386 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:09:59.930341 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:01.188086 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:01.188047 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:01.457485 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:01.457363 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:03.950439 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:03.950389 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:04.154170 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:04.154136 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:04.353004 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:04.352973 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:06.167719 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:06.167681 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:07.512565 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:07.512528 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:08.884544 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:08.884506 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:09.115853 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:09.115815 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:09.325444 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:09.325372 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:09.526390 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:09.526347 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:09.741888 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:09.741807 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:13.358541 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:13.358430 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:13.572389 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:13.570054 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:18.085221 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:18.085177 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:22.676738 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:22.676685 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:26.239816 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:10:26.239779 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:10:27.110157 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:27.110116 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:31.567184 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:31.567079 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:34.941255 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:34.941218 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:35.147708 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:35.147675 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:36.024811 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:36.024769 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:40.360488 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:40.359117 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:40.568681 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:40.568646 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:40.768577 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:40.768487 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:44.992368 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:44.992323 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:49.550804 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:49.550718 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:54.079102 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:54.079052 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:56.237263 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:10:56.237226 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:10:58.566096 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:58.566056 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:58.772153 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:58.772093 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:10:58.976466 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:10:58.976365 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:03.220900 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:03.220873 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 18:11:03.222218 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:03.222195 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 18:11:03.695097 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:03.695062 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:03.897858 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:03.897825 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:04.139150 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:04.139111 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:04.878947 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:04.878906 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:05.107508 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:05.107472 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:08.703686 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:08.703651 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:10.213287 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:10.213252 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:10.423664 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:10.423631 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:12.461060 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:12.461021 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:13.673607 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:13.673563 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:13.877710 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:13.877676 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:14.752964 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:14.752929 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:14.951199 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:14.951125 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:18.511948 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:18.511906 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:23.331417 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:23.326033 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:26.237006 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:11:26.236966 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:11:26.882187 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:26.882150 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:27.086450 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:27.086420 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:27.432006 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:27.431969 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:28.087745 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:28.087702 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:28.296070 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:28.296021 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:28.498618 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:28.498526 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:29.049482 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:29.049439 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fd8d1c6-xgqjv"] Nov 03 18:11:29.095568 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:29.095531 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fd8d1c6-xgqjv"] Nov 03 18:11:30.977177 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:30.977144 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8" path="/var/lib/kubelet/pods/3e0e7d5a-927e-4de3-af8e-fdc1aacbc3a8/volumes" Nov 03 18:11:32.889663 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:32.889628 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:33.356269 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:33.356230 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:33.563660 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:33.563626 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:36.544244 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:36.544209 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:36.754806 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:36.754756 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:37.231357 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:37.231321 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h"] Nov 03 18:11:37.235924 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:37.235896 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h" Nov 03 18:11:37.285023 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:37.284975 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-7a7c76\"" Nov 03 18:11:37.290369 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:37.290326 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h"] Nov 03 18:11:37.331101 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:37.331068 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-7a7c76-dockercfg-xd5h7\"" Nov 03 18:11:37.355434 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:37.355381 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b2f655-b26e-4a96-a114-5596317d889e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h\" (UID: \"36b2f655-b26e-4a96-a114-5596317d889e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h" Nov 03 18:11:37.355708 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:37.355681 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq795\" (UniqueName: \"kubernetes.io/projected/36b2f655-b26e-4a96-a114-5596317d889e-kube-api-access-hq795\") pod \"payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h\" (UID: \"36b2f655-b26e-4a96-a114-5596317d889e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h" Nov 03 18:11:37.457105 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:37.457067 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq795\" (UniqueName: \"kubernetes.io/projected/36b2f655-b26e-4a96-a114-5596317d889e-kube-api-access-hq795\") pod \"payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h\" (UID: \"36b2f655-b26e-4a96-a114-5596317d889e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h" Nov 03 18:11:37.457316 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:37.457235 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b2f655-b26e-4a96-a114-5596317d889e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h\" (UID: \"36b2f655-b26e-4a96-a114-5596317d889e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h" Nov 03 18:11:37.460462 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:37.458351 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b2f655-b26e-4a96-a114-5596317d889e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h\" (UID: \"36b2f655-b26e-4a96-a114-5596317d889e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h" Nov 03 18:11:37.515207 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:37.515136 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq795\" (UniqueName: \"kubernetes.io/projected/36b2f655-b26e-4a96-a114-5596317d889e-kube-api-access-hq795\") pod \"payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h\" (UID: \"36b2f655-b26e-4a96-a114-5596317d889e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h" Nov 03 18:11:37.553738 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:37.553702 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h" Nov 03 18:11:37.913770 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:37.913732 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:37.981848 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:11:37.981814 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b2f655_b26e_4a96_a114_5596317d889e.slice/crio-e4cac485a5c24543c35193b13527e0dd433a090e16ff61169bcd02448aab9f13 WatchSource:0}: Error finding container e4cac485a5c24543c35193b13527e0dd433a090e16ff61169bcd02448aab9f13: Status 404 returned error can't find the container with id e4cac485a5c24543c35193b13527e0dd433a090e16ff61169bcd02448aab9f13 Nov 03 18:11:37.982130 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:37.982104 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h"] Nov 03 18:11:38.275983 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:38.275891 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h" event={"ID":"36b2f655-b26e-4a96-a114-5596317d889e","Type":"ContainerStarted","Data":"5e31b7160f8a711f78caf347ce5e5c76e4de0bfa5e438d32d278f06c299e801d"} Nov 03 18:11:38.275983 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:38.275948 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h" event={"ID":"36b2f655-b26e-4a96-a114-5596317d889e","Type":"ContainerStarted","Data":"e4cac485a5c24543c35193b13527e0dd433a090e16ff61169bcd02448aab9f13"} Nov 03 18:11:38.276213 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:38.276132 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h" Nov 03 18:11:38.387049 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:38.386994 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h" podStartSLOduration=1.38698013 podStartE2EDuration="1.38698013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:11:38.383560457 +0000 UTC m=+1535.517189467" watchObservedRunningTime="2026-11-03 18:11:38.38698013 +0000 UTC m=+1535.520609158" Nov 03 18:11:41.558525 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:41.558490 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:42.855676 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:42.855635 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:47.742460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:47.742418 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:47.944414 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:47.944370 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:48.147929 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:48.147892 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:49.019364 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:49.019329 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:49.218995 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:49.218957 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:49.285500 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:49.285387 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h" Nov 03 18:11:49.700637 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:49.700603 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:51.230712 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:51.230678 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:51.430811 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:51.430772 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:52.951048 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:52.950985 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:56.236719 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:11:56.236678 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:11:56.265413 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:56.265366 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:11:57.435895 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:11:57.435846 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:01.333362 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:01.333324 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:02.803120 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:02.803082 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:03.013170 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:03.013130 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:03.260615 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:03.260573 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:03.476601 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:03.476568 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:03.693220 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:03.691162 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:04.275180 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:04.275146 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:04.973498 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:04.973457 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:05.208459 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:05.193310 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:08.062114 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:08.062021 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:09.548348 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:09.548271 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:10.049079 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:10.049035 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:12.585236 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:12.585172 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:12.893713 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:12.893621 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:14.047615 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:14.047576 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:15.276792 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:15.276760 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p"] Nov 03 18:12:15.277196 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:15.276961 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" podUID="86d8d1e1-cbb6-4e0f-93ac-9a19035e7681" containerName="istio-proxy" containerID="cri-o://8df3f84577c058838bacc55be4aed36b380468a334031e3bad56e43191e683af" gracePeriod=30 Nov 03 18:12:15.322476 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:15.322442 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77"] Nov 03 18:12:15.322723 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:15.322692 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" podUID="0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" containerName="istio-proxy" containerID="cri-o://06e131755902e8c425619882fb0456acde935b06267c090ead3ca977abd0d624" gracePeriod=30 Nov 03 18:12:15.328440 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:15.328414 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f"] Nov 03 18:12:15.328649 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:15.328627 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" podUID="aca6224c-f93a-4d51-b37e-f3368efa25dd" containerName="istio-proxy" containerID="cri-o://8e5cf7007df00719ab605d265227147d1397bb6ce550f9ce96c3492d4bcb91c4" gracePeriod=30 Nov 03 18:12:15.368228 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:15.368195 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96"] Nov 03 18:12:15.368453 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:15.368430 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" podUID="2df51a82-bd64-4ad8-a164-d5af9ee53e2d" containerName="istio-proxy" containerID="cri-o://d7a7f658b2d633e3ae1325702d98b3cede880463b91edc9d73a345404110fb20" gracePeriod=30 Nov 03 18:12:17.382145 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:17.382106 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:17.913020 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:17.912983 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:18.121146 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:18.121090 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:19.264494 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:19.264461 5805 patch_prober.go:29] interesting pod/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:12:19.264980 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:19.264516 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" podUID="86d8d1e1-cbb6-4e0f-93ac-9a19035e7681" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:12:20.662953 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.662367 5805 generic.go:356] "Generic (PLEG): container finished" podID="2df51a82-bd64-4ad8-a164-d5af9ee53e2d" containerID="d7a7f658b2d633e3ae1325702d98b3cede880463b91edc9d73a345404110fb20" exitCode=0 Nov 03 18:12:20.662953 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.662606 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" event={"ID":"2df51a82-bd64-4ad8-a164-d5af9ee53e2d","Type":"ContainerDied","Data":"d7a7f658b2d633e3ae1325702d98b3cede880463b91edc9d73a345404110fb20"} Nov 03 18:12:20.667611 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.667011 5805 generic.go:356] "Generic (PLEG): container finished" podID="86d8d1e1-cbb6-4e0f-93ac-9a19035e7681" containerID="8df3f84577c058838bacc55be4aed36b380468a334031e3bad56e43191e683af" exitCode=0 Nov 03 18:12:20.667611 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.667133 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" event={"ID":"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681","Type":"ContainerDied","Data":"8df3f84577c058838bacc55be4aed36b380468a334031e3bad56e43191e683af"} Nov 03 18:12:20.669577 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.669522 5805 generic.go:356] "Generic (PLEG): container finished" podID="0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" containerID="06e131755902e8c425619882fb0456acde935b06267c090ead3ca977abd0d624" exitCode=0 Nov 03 18:12:20.669760 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.669547 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" event={"ID":"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac","Type":"ContainerDied","Data":"06e131755902e8c425619882fb0456acde935b06267c090ead3ca977abd0d624"} Nov 03 18:12:20.672946 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.672796 5805 generic.go:356] "Generic (PLEG): container finished" podID="aca6224c-f93a-4d51-b37e-f3368efa25dd" containerID="8e5cf7007df00719ab605d265227147d1397bb6ce550f9ce96c3492d4bcb91c4" exitCode=0 Nov 03 18:12:20.672946 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.672866 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" event={"ID":"aca6224c-f93a-4d51-b37e-f3368efa25dd","Type":"ContainerDied","Data":"8e5cf7007df00719ab605d265227147d1397bb6ce550f9ce96c3492d4bcb91c4"} Nov 03 18:12:20.756502 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.755446 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:12:20.833384 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.833154 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:12:20.837326 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.837248 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:12:20.843855 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.843830 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istiod-ca-cert\") pod \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " Nov 03 18:12:20.844003 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.843894 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-data\") pod \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " Nov 03 18:12:20.844003 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.843918 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-envoy\") pod \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " Nov 03 18:12:20.844003 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.843941 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-token\" (UniqueName: \"kubernetes.io/projected/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-token\") pod \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " Nov 03 18:12:20.844003 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.843968 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-credential-socket\") pod \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " Nov 03 18:12:20.844226 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.844019 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-kube-api-access-mncjm\" (UniqueName: \"kubernetes.io/projected/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-kube-api-access-mncjm\") pod \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " Nov 03 18:12:20.844226 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.844186 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-workload-certs\") pod \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " Nov 03 18:12:20.844226 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.844219 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-workload-socket\") pod \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " Nov 03 18:12:20.844385 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.844246 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-podinfo\") pod \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\" (UID: \"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681\") " Nov 03 18:12:20.845460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.845294 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istiod-ca-cert" pod "86d8d1e1-cbb6-4e0f-93ac-9a19035e7681" (UID: "86d8d1e1-cbb6-4e0f-93ac-9a19035e7681"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:12:20.845583 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.845513 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-data" pod "86d8d1e1-cbb6-4e0f-93ac-9a19035e7681" (UID: "86d8d1e1-cbb6-4e0f-93ac-9a19035e7681"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:20.845646 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.845627 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-workload-certs" pod "86d8d1e1-cbb6-4e0f-93ac-9a19035e7681" (UID: "86d8d1e1-cbb6-4e0f-93ac-9a19035e7681"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:20.847050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.846953 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-credential-socket" pod "86d8d1e1-cbb6-4e0f-93ac-9a19035e7681" (UID: "86d8d1e1-cbb6-4e0f-93ac-9a19035e7681"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:20.847050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.847025 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-workload-socket" pod "86d8d1e1-cbb6-4e0f-93ac-9a19035e7681" (UID: "86d8d1e1-cbb6-4e0f-93ac-9a19035e7681"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:20.851585 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.851561 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-podinfo" pod "86d8d1e1-cbb6-4e0f-93ac-9a19035e7681" (UID: "86d8d1e1-cbb6-4e0f-93ac-9a19035e7681"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:12:20.852063 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.852035 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-token" pod "86d8d1e1-cbb6-4e0f-93ac-9a19035e7681" (UID: "86d8d1e1-cbb6-4e0f-93ac-9a19035e7681"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:12:20.856387 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.855753 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-envoy" pod "86d8d1e1-cbb6-4e0f-93ac-9a19035e7681" (UID: "86d8d1e1-cbb6-4e0f-93ac-9a19035e7681"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:20.857153 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.857131 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-kube-api-access-mncjm" pod "86d8d1e1-cbb6-4e0f-93ac-9a19035e7681" (UID: "86d8d1e1-cbb6-4e0f-93ac-9a19035e7681"). InnerVolumeSpecName "kube-api-access-mncjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:12:20.881407 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.881379 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:12:20.945502 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.945474 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-podinfo\") pod \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " Nov 03 18:12:20.945664 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.945531 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-data\") pod \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " Nov 03 18:12:20.945664 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.945574 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-workload-certs\") pod \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " Nov 03 18:12:20.945664 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.945626 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-envoy\") pod \"aca6224c-f93a-4d51-b37e-f3368efa25dd\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " Nov 03 18:12:20.945838 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.945692 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istiod-ca-cert\") pod \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " Nov 03 18:12:20.945838 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.945728 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aca6224c-f93a-4d51-b37e-f3368efa25dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca6224c-f93a-4d51-b37e-f3368efa25dd-istiod-ca-cert\") pod \"aca6224c-f93a-4d51-b37e-f3368efa25dd\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " Nov 03 18:12:20.945838 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.945785 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-workload-socket\") pod \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " Nov 03 18:12:20.945838 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.945820 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-token\" (UniqueName: \"kubernetes.io/projected/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-token\") pod \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " Nov 03 18:12:20.946044 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.945854 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-credential-socket\") pod \"aca6224c-f93a-4d51-b37e-f3368efa25dd\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " Nov 03 18:12:20.946044 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.945874 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-data" pod "0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" (UID: "0133a1d6-2676-4f88-8fee-1dfe9c2b28ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:20.946162 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.946053 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-credential-socket" pod "aca6224c-f93a-4d51-b37e-f3368efa25dd" (UID: "aca6224c-f93a-4d51-b37e-f3368efa25dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:20.946162 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.946063 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-workload-socket" pod "0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" (UID: "0133a1d6-2676-4f88-8fee-1dfe9c2b28ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:20.946162 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.946141 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istiod-ca-cert" pod "0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" (UID: "0133a1d6-2676-4f88-8fee-1dfe9c2b28ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:12:20.946325 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.946192 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-token\" (UniqueName: \"kubernetes.io/projected/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-token\") pod \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " Nov 03 18:12:20.946428 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.946390 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-workload-socket\") pod \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " Nov 03 18:12:20.946498 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.946415 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aca6224c-f93a-4d51-b37e-f3368efa25dd-istiod-ca-cert" pod "aca6224c-f93a-4d51-b37e-f3368efa25dd" (UID: "aca6224c-f93a-4d51-b37e-f3368efa25dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:12:20.946498 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.946459 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-workload-certs\") pod \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " Nov 03 18:12:20.946498 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.946486 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-podinfo\") pod \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " Nov 03 18:12:20.946652 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.946551 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-credential-socket\") pod \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " Nov 03 18:12:20.946652 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.946582 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-envoy\") pod \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " Nov 03 18:12:20.946652 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.946613 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-kube-api-access-72c24\" (UniqueName: \"kubernetes.io/projected/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-kube-api-access-72c24\") pod \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " Nov 03 18:12:20.946804 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.946662 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-podinfo\") pod \"aca6224c-f93a-4d51-b37e-f3368efa25dd\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " Nov 03 18:12:20.946804 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.946706 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-envoy\") pod \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " Nov 03 18:12:20.946804 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.946753 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca6224c-f93a-4d51-b37e-f3368efa25dd-kube-api-access-5zwhx\" (UniqueName: \"kubernetes.io/projected/aca6224c-f93a-4d51-b37e-f3368efa25dd-kube-api-access-5zwhx\") pod \"aca6224c-f93a-4d51-b37e-f3368efa25dd\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " Nov 03 18:12:20.947011 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.946808 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-token\" (UniqueName: \"kubernetes.io/projected/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-token\") pod \"aca6224c-f93a-4d51-b37e-f3368efa25dd\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " Nov 03 18:12:20.947011 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.946835 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-data\") pod \"aca6224c-f93a-4d51-b37e-f3368efa25dd\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " Nov 03 18:12:20.947011 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.946868 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-credential-socket\") pod \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\" (UID: \"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac\") " Nov 03 18:12:20.947011 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.946941 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-kube-api-access-tlsdc\" (UniqueName: \"kubernetes.io/projected/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-kube-api-access-tlsdc\") pod \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " Nov 03 18:12:20.947011 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.946992 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-data\") pod \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " Nov 03 18:12:20.947793 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.947028 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istiod-ca-cert\") pod \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\" (UID: \"2df51a82-bd64-4ad8-a164-d5af9ee53e2d\") " Nov 03 18:12:20.947793 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.947073 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-workload-certs\") pod \"aca6224c-f93a-4d51-b37e-f3368efa25dd\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " Nov 03 18:12:20.947793 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.947169 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-workload-socket\") pod \"aca6224c-f93a-4d51-b37e-f3368efa25dd\" (UID: \"aca6224c-f93a-4d51-b37e-f3368efa25dd\") " Nov 03 18:12:20.949050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.948318 5805 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-workload-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:20.949050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.948349 5805 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-data\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:20.949050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.948350 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-workload-certs" pod "2df51a82-bd64-4ad8-a164-d5af9ee53e2d" (UID: "2df51a82-bd64-4ad8-a164-d5af9ee53e2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:20.949050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.948365 5805 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-workload-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:20.949050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.948380 5805 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-podinfo\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:20.949050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.948407 5805 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istiod-ca-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:20.949050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.948421 5805 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca6224c-f93a-4d51-b37e-f3368efa25dd-istiod-ca-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:20.949050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.948435 5805 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istiod-ca-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:20.949050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.948450 5805 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-workload-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:20.949050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.948464 5805 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-credential-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:20.949050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.948477 5805 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-data\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:20.949050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.948492 5805 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-envoy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:20.949050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.948506 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:20.949050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.948520 5805 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-credential-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:20.949050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.948534 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mncjm\" (UniqueName: \"kubernetes.io/projected/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681-kube-api-access-mncjm\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:20.949050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.948597 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-credential-socket" pod "2df51a82-bd64-4ad8-a164-d5af9ee53e2d" (UID: "2df51a82-bd64-4ad8-a164-d5af9ee53e2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:20.949050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.948727 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-workload-socket" pod "aca6224c-f93a-4d51-b37e-f3368efa25dd" (UID: "aca6224c-f93a-4d51-b37e-f3368efa25dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:20.949050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.948783 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-workload-socket" pod "2df51a82-bd64-4ad8-a164-d5af9ee53e2d" (UID: "2df51a82-bd64-4ad8-a164-d5af9ee53e2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:20.949050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.948751 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-envoy" pod "aca6224c-f93a-4d51-b37e-f3368efa25dd" (UID: "aca6224c-f93a-4d51-b37e-f3368efa25dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:20.949050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.948978 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-workload-certs" pod "0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" (UID: "0133a1d6-2676-4f88-8fee-1dfe9c2b28ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:20.950121 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.949494 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-token" pod "0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" (UID: "0133a1d6-2676-4f88-8fee-1dfe9c2b28ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:12:20.950121 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.949751 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-data" pod "aca6224c-f93a-4d51-b37e-f3368efa25dd" (UID: "aca6224c-f93a-4d51-b37e-f3368efa25dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:20.950617 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.950592 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-podinfo" pod "0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" (UID: "0133a1d6-2676-4f88-8fee-1dfe9c2b28ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:12:20.950909 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.950879 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-envoy" pod "2df51a82-bd64-4ad8-a164-d5af9ee53e2d" (UID: "2df51a82-bd64-4ad8-a164-d5af9ee53e2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:20.951149 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.951127 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-credential-socket" pod "0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" (UID: "0133a1d6-2676-4f88-8fee-1dfe9c2b28ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:20.952428 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.952384 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istiod-ca-cert" pod "2df51a82-bd64-4ad8-a164-d5af9ee53e2d" (UID: "2df51a82-bd64-4ad8-a164-d5af9ee53e2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:12:20.952763 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.952741 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-workload-certs" pod "aca6224c-f93a-4d51-b37e-f3368efa25dd" (UID: "aca6224c-f93a-4d51-b37e-f3368efa25dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:20.953039 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.953010 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-data" pod "2df51a82-bd64-4ad8-a164-d5af9ee53e2d" (UID: "2df51a82-bd64-4ad8-a164-d5af9ee53e2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:20.953132 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.953035 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-podinfo" pod "aca6224c-f93a-4d51-b37e-f3368efa25dd" (UID: "aca6224c-f93a-4d51-b37e-f3368efa25dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:12:20.953346 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.953264 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-podinfo" pod "2df51a82-bd64-4ad8-a164-d5af9ee53e2d" (UID: "2df51a82-bd64-4ad8-a164-d5af9ee53e2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:12:20.953346 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.953281 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-kube-api-access-tlsdc" pod "2df51a82-bd64-4ad8-a164-d5af9ee53e2d" (UID: "2df51a82-bd64-4ad8-a164-d5af9ee53e2d"). InnerVolumeSpecName "kube-api-access-tlsdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:12:20.953346 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.953327 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca6224c-f93a-4d51-b37e-f3368efa25dd-kube-api-access-5zwhx" pod "aca6224c-f93a-4d51-b37e-f3368efa25dd" (UID: "aca6224c-f93a-4d51-b37e-f3368efa25dd"). InnerVolumeSpecName "kube-api-access-5zwhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:12:20.953522 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.953416 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-kube-api-access-72c24" pod "0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" (UID: "0133a1d6-2676-4f88-8fee-1dfe9c2b28ac"). InnerVolumeSpecName "kube-api-access-72c24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:12:20.953563 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.953513 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-token" pod "2df51a82-bd64-4ad8-a164-d5af9ee53e2d" (UID: "2df51a82-bd64-4ad8-a164-d5af9ee53e2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:12:20.953998 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.953973 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-token" pod "aca6224c-f93a-4d51-b37e-f3368efa25dd" (UID: "aca6224c-f93a-4d51-b37e-f3368efa25dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:12:20.954983 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:20.954963 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-envoy" pod "0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" (UID: "0133a1d6-2676-4f88-8fee-1dfe9c2b28ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:21.050095 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.050066 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72c24\" (UniqueName: \"kubernetes.io/projected/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-kube-api-access-72c24\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:21.050361 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.050322 5805 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-podinfo\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:21.050361 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.050360 5805 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-envoy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:21.050526 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.050376 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zwhx\" (UniqueName: \"kubernetes.io/projected/aca6224c-f93a-4d51-b37e-f3368efa25dd-kube-api-access-5zwhx\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:21.050526 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.050389 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:21.050526 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.050417 5805 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-data\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:21.050526 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.050430 5805 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-credential-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:21.050526 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.050444 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlsdc\" (UniqueName: \"kubernetes.io/projected/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-kube-api-access-tlsdc\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:21.050526 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.050457 5805 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-data\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:21.050526 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.050471 5805 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istiod-ca-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:21.050526 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.050484 5805 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-workload-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:21.050526 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.050492 5805 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-workload-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:21.050526 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.050501 5805 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-podinfo\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:21.050526 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.050510 5805 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-workload-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:21.050526 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.050518 5805 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca6224c-f93a-4d51-b37e-f3368efa25dd-istio-envoy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:21.050526 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.050527 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:21.050905 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.050539 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:21.050905 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.050548 5805 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-workload-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:21.050905 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.050556 5805 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-workload-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:21.050905 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.050565 5805 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-istio-podinfo\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:21.050905 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.050573 5805 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2df51a82-bd64-4ad8-a164-d5af9ee53e2d-credential-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:21.050905 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.050581 5805 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac-istio-envoy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:21.682177 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.682094 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" Nov 03 18:12:21.682632 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.682090 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p" event={"ID":"86d8d1e1-cbb6-4e0f-93ac-9a19035e7681","Type":"ContainerDied","Data":"689f9d68b8118ea201e4588bf0d629465be1d38a02af66354009b8f09eae8324"} Nov 03 18:12:21.682632 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.682226 5805 scope.go:122] "RemoveContainer" containerID="8df3f84577c058838bacc55be4aed36b380468a334031e3bad56e43191e683af" Nov 03 18:12:21.683779 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.683746 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" Nov 03 18:12:21.683779 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.683769 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77" event={"ID":"0133a1d6-2676-4f88-8fee-1dfe9c2b28ac","Type":"ContainerDied","Data":"340e5847007adcd59f97c6a84d6588c03de5d1cb8c0c50e40e93f687efbddc49"} Nov 03 18:12:21.685344 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.685322 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" Nov 03 18:12:21.685466 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.685372 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f" event={"ID":"aca6224c-f93a-4d51-b37e-f3368efa25dd","Type":"ContainerDied","Data":"8c168f0495eac16288c9faa0960aa0a95ba824941afbd985722167d9c50afac5"} Nov 03 18:12:21.687003 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.686984 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" Nov 03 18:12:21.687123 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.687008 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96" event={"ID":"2df51a82-bd64-4ad8-a164-d5af9ee53e2d","Type":"ContainerDied","Data":"6bdaee003e362c17415496afeaaa9df90313ac27b7534c1da566e93756d28b05"} Nov 03 18:12:21.701739 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.701720 5805 scope.go:122] "RemoveContainer" containerID="06e131755902e8c425619882fb0456acde935b06267c090ead3ca977abd0d624" Nov 03 18:12:21.710885 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.710867 5805 scope.go:122] "RemoveContainer" containerID="8e5cf7007df00719ab605d265227147d1397bb6ce550f9ce96c3492d4bcb91c4" Nov 03 18:12:21.720202 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.720182 5805 scope.go:122] "RemoveContainer" containerID="d7a7f658b2d633e3ae1325702d98b3cede880463b91edc9d73a345404110fb20" Nov 03 18:12:21.838794 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.838702 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p"] Nov 03 18:12:21.885253 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.885224 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-s875p"] Nov 03 18:12:21.978206 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.978108 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:21.994061 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.994030 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77"] Nov 03 18:12:21.997466 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:21.997443 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-rcf77"] Nov 03 18:12:22.095865 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:22.095835 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f"] Nov 03 18:12:22.102465 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:22.102387 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-9m85f"] Nov 03 18:12:22.205171 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:22.205132 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96"] Nov 03 18:12:22.208936 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:22.208903 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aab753-openshift-default-588c6d4c79-h2c96"] Nov 03 18:12:22.977622 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:22.977580 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" path="/var/lib/kubelet/pods/0133a1d6-2676-4f88-8fee-1dfe9c2b28ac/volumes" Nov 03 18:12:22.978269 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:22.978247 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2df51a82-bd64-4ad8-a164-d5af9ee53e2d" path="/var/lib/kubelet/pods/2df51a82-bd64-4ad8-a164-d5af9ee53e2d/volumes" Nov 03 18:12:22.978787 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:22.978765 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86d8d1e1-cbb6-4e0f-93ac-9a19035e7681" path="/var/lib/kubelet/pods/86d8d1e1-cbb6-4e0f-93ac-9a19035e7681/volumes" Nov 03 18:12:22.979579 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:22.979559 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aca6224c-f93a-4d51-b37e-f3368efa25dd" path="/var/lib/kubelet/pods/aca6224c-f93a-4d51-b37e-f3368efa25dd/volumes" Nov 03 18:12:26.236070 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:12:26.236034 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:12:26.467024 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:26.466987 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:30.055302 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:30.055267 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8"] Nov 03 18:12:30.055721 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:30.055522 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" podUID="0b450130-5eea-4fe2-896c-62ba2fe84807" containerName="istio-proxy" containerID="cri-o://8d424cce82c533303c9f10ff5040190a28b0ee508202d0bba49d20527c0b4094" gracePeriod=30 Nov 03 18:12:30.814717 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:30.814678 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:31.024898 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:31.024865 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:31.243683 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:31.243602 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:35.761389 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.761364 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:12:35.826627 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.826540 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:35.855320 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.855285 5805 generic.go:356] "Generic (PLEG): container finished" podID="0b450130-5eea-4fe2-896c-62ba2fe84807" containerID="8d424cce82c533303c9f10ff5040190a28b0ee508202d0bba49d20527c0b4094" exitCode=0 Nov 03 18:12:35.855532 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.855337 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" Nov 03 18:12:35.855532 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.855370 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" event={"ID":"0b450130-5eea-4fe2-896c-62ba2fe84807","Type":"ContainerDied","Data":"8d424cce82c533303c9f10ff5040190a28b0ee508202d0bba49d20527c0b4094"} Nov 03 18:12:35.855532 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.855450 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8" event={"ID":"0b450130-5eea-4fe2-896c-62ba2fe84807","Type":"ContainerDied","Data":"130f667332f9c8cad93e8910a70705d2708c0af50dc5cef01111ab08c4415570"} Nov 03 18:12:35.855532 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.855484 5805 scope.go:122] "RemoveContainer" containerID="8d424cce82c533303c9f10ff5040190a28b0ee508202d0bba49d20527c0b4094" Nov 03 18:12:35.865511 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.865476 5805 scope.go:122] "RemoveContainer" containerID="8d424cce82c533303c9f10ff5040190a28b0ee508202d0bba49d20527c0b4094" Nov 03 18:12:35.865762 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:12:35.865744 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d424cce82c533303c9f10ff5040190a28b0ee508202d0bba49d20527c0b4094\": container with ID starting with 8d424cce82c533303c9f10ff5040190a28b0ee508202d0bba49d20527c0b4094 not found: ID does not exist" containerID="8d424cce82c533303c9f10ff5040190a28b0ee508202d0bba49d20527c0b4094" Nov 03 18:12:35.865810 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.865775 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d424cce82c533303c9f10ff5040190a28b0ee508202d0bba49d20527c0b4094"} err="failed to get container status \"8d424cce82c533303c9f10ff5040190a28b0ee508202d0bba49d20527c0b4094\": rpc error: code = NotFound desc = could not find container \"8d424cce82c533303c9f10ff5040190a28b0ee508202d0bba49d20527c0b4094\": container with ID starting with 8d424cce82c533303c9f10ff5040190a28b0ee508202d0bba49d20527c0b4094 not found: ID does not exist" Nov 03 18:12:35.904294 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.904252 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-workload-certs\") pod \"0b450130-5eea-4fe2-896c-62ba2fe84807\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " Nov 03 18:12:35.904494 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.904470 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-envoy\") pod \"0b450130-5eea-4fe2-896c-62ba2fe84807\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " Nov 03 18:12:35.904494 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.904476 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-workload-certs" pod "0b450130-5eea-4fe2-896c-62ba2fe84807" (UID: "0b450130-5eea-4fe2-896c-62ba2fe84807"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:35.904698 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.904533 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b450130-5eea-4fe2-896c-62ba2fe84807-kube-api-access-9n4s8\" (UniqueName: \"kubernetes.io/projected/0b450130-5eea-4fe2-896c-62ba2fe84807-kube-api-access-9n4s8\") pod \"0b450130-5eea-4fe2-896c-62ba2fe84807\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " Nov 03 18:12:35.904698 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.904610 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-workload-socket\") pod \"0b450130-5eea-4fe2-896c-62ba2fe84807\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " Nov 03 18:12:35.904698 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.904636 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b450130-5eea-4fe2-896c-62ba2fe84807-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b450130-5eea-4fe2-896c-62ba2fe84807-istiod-ca-cert\") pod \"0b450130-5eea-4fe2-896c-62ba2fe84807\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " Nov 03 18:12:35.904698 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.904670 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-credential-socket\") pod \"0b450130-5eea-4fe2-896c-62ba2fe84807\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " Nov 03 18:12:35.904906 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.904721 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-podinfo\") pod \"0b450130-5eea-4fe2-896c-62ba2fe84807\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " Nov 03 18:12:35.905019 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.904983 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-credential-socket" pod "0b450130-5eea-4fe2-896c-62ba2fe84807" (UID: "0b450130-5eea-4fe2-896c-62ba2fe84807"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:35.905165 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.905144 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b450130-5eea-4fe2-896c-62ba2fe84807-istiod-ca-cert" pod "0b450130-5eea-4fe2-896c-62ba2fe84807" (UID: "0b450130-5eea-4fe2-896c-62ba2fe84807"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:12:35.905324 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.905300 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-workload-socket" pod "0b450130-5eea-4fe2-896c-62ba2fe84807" (UID: "0b450130-5eea-4fe2-896c-62ba2fe84807"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:35.905461 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.905443 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-data\") pod \"0b450130-5eea-4fe2-896c-62ba2fe84807\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " Nov 03 18:12:35.905557 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.905502 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-token\" (UniqueName: \"kubernetes.io/projected/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-token\") pod \"0b450130-5eea-4fe2-896c-62ba2fe84807\" (UID: \"0b450130-5eea-4fe2-896c-62ba2fe84807\") " Nov 03 18:12:35.905729 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.905704 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-data" pod "0b450130-5eea-4fe2-896c-62ba2fe84807" (UID: "0b450130-5eea-4fe2-896c-62ba2fe84807"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:35.906223 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.906199 5805 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-workload-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:35.906223 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.906229 5805 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b450130-5eea-4fe2-896c-62ba2fe84807-istiod-ca-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:35.906390 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.906247 5805 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-credential-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:35.906390 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.906263 5805 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-data\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:35.906390 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.906280 5805 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-workload-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:35.906965 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.906927 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b450130-5eea-4fe2-896c-62ba2fe84807-kube-api-access-9n4s8" pod "0b450130-5eea-4fe2-896c-62ba2fe84807" (UID: "0b450130-5eea-4fe2-896c-62ba2fe84807"). InnerVolumeSpecName "kube-api-access-9n4s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:12:35.907095 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.907074 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-envoy" pod "0b450130-5eea-4fe2-896c-62ba2fe84807" (UID: "0b450130-5eea-4fe2-896c-62ba2fe84807"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:35.907379 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.907359 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-podinfo" pod "0b450130-5eea-4fe2-896c-62ba2fe84807" (UID: "0b450130-5eea-4fe2-896c-62ba2fe84807"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:12:35.907733 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:35.907716 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-token" pod "0b450130-5eea-4fe2-896c-62ba2fe84807" (UID: "0b450130-5eea-4fe2-896c-62ba2fe84807"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:12:36.012048 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:36.012009 5805 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-podinfo\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:36.012048 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:36.012046 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:36.012048 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:36.012061 5805 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b450130-5eea-4fe2-896c-62ba2fe84807-istio-envoy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:36.012048 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:36.012075 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n4s8\" (UniqueName: \"kubernetes.io/projected/0b450130-5eea-4fe2-896c-62ba2fe84807-kube-api-access-9n4s8\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:12:36.275741 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:36.275704 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8"] Nov 03 18:12:36.284250 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:36.284214 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-8dwj8"] Nov 03 18:12:36.587282 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:36.587250 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:36.978274 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:36.978206 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b450130-5eea-4fe2-896c-62ba2fe84807" path="/var/lib/kubelet/pods/0b450130-5eea-4fe2-896c-62ba2fe84807/volumes" Nov 03 18:12:40.453979 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:40.453939 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:43.345672 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:43.345637 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:45.074715 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:45.074668 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:49.634256 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:49.634223 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:49.842944 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:49.842909 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:54.079217 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:54.079178 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:12:56.235325 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:12:56.235288 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:12:58.582831 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:12:58.582793 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:03.082932 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:03.082887 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:03.264280 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:03.264246 5805 scope.go:122] "RemoveContainer" containerID="e1801f664c252265ca4db55fa3df725af5ad5f8ff3dd6fe0e33f72ee77e47605" Nov 03 18:13:07.621212 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:07.621178 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:07.881766 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:07.881667 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:08.419245 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:08.419209 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:12.405343 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:12.405250 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:12.603600 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:12.603564 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:14.380813 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:14.380781 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:16.971461 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:16.971380 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:21.735679 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:21.735636 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:26.236835 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:13:26.236796 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:13:26.530532 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:26.530370 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:28.942315 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:28.942274 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:31.376201 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:31.376166 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:33.534828 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:33.532927 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:33.732486 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:33.732452 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:33.845067 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:33.845022 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52"] Nov 03 18:13:33.890869 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:33.890832 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8"] Nov 03 18:13:34.059184 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:34.059154 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:36.294782 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:36.294736 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:38.577715 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:38.577671 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:38.801841 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:38.801790 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:41.183618 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:41.183583 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:43.954366 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:43.953997 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:44.156099 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:44.156059 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:45.237740 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:45.237707 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:47.721437 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:47.721373 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:47.930896 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:47.930856 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:48.256438 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:48.256345 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:48.464629 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:48.464593 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:48.669342 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:48.669302 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:48.867484 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:48.867442 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" podUID="2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" containerName="main" containerID="cri-o://b639272e70bb94c33b737d2f51c99472cd6928386466bd3dd69ddbc557ecbc66" gracePeriod=45 Nov 03 18:13:48.912307 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:48.912266 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" containerID="cri-o://5e603a64e0fa63996c250c63a20f8e96e690b8ffa7220370c0cea00a274492fe" gracePeriod=45 Nov 03 18:13:49.250673 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.250647 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:13:49.254662 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.254639 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 18:13:49.321303 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.319176 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-model-cache\") pod \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\" (UID: \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\") " Nov 03 18:13:49.321303 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.319249 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-dshm\") pod \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\" (UID: \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\") " Nov 03 18:13:49.321303 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.319308 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-tmp-dir\") pod \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\" (UID: \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\") " Nov 03 18:13:49.321303 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.319356 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-kube-api-access-nlfm2\" (UniqueName: \"kubernetes.io/projected/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-kube-api-access-nlfm2\") pod \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\" (UID: \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\") " Nov 03 18:13:49.321303 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.319441 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-home\" (UniqueName: \"kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-home\") pod \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\" (UID: \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\") " Nov 03 18:13:49.321303 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.319508 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-model-cache" pod "2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" (UID: "2353fc1e-3b17-41f1-a8f4-47c2bc2cc260"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:49.321303 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.319541 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-tls-certs\" (UniqueName: \"kubernetes.io/secret/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-tls-certs\") pod \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\" (UID: \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\") " Nov 03 18:13:49.321303 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.319609 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-dshm\") pod \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\" (UID: \"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260\") " Nov 03 18:13:49.321303 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.319657 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43eecb04-1091-4dde-9b9c-42a0fb28f08f-kube-api-access-jj2pv\" (UniqueName: \"kubernetes.io/projected/43eecb04-1091-4dde-9b9c-42a0fb28f08f-kube-api-access-jj2pv\") pod \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\" (UID: \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\") " Nov 03 18:13:49.321303 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.319682 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-tmp-dir" pod "2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" (UID: "2353fc1e-3b17-41f1-a8f4-47c2bc2cc260"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:49.321303 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.319729 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43eecb04-1091-4dde-9b9c-42a0fb28f08f-tls-certs\" (UniqueName: \"kubernetes.io/secret/43eecb04-1091-4dde-9b9c-42a0fb28f08f-tls-certs\") pod \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\" (UID: \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\") " Nov 03 18:13:49.321303 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.319807 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-tmp-dir\") pod \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\" (UID: \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\") " Nov 03 18:13:49.321303 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.319879 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-model-cache\") pod \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\" (UID: \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\") " Nov 03 18:13:49.321303 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.319931 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-home\" (UniqueName: \"kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-home\") pod \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\" (UID: \"43eecb04-1091-4dde-9b9c-42a0fb28f08f\") " Nov 03 18:13:49.321303 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.320658 5805 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-model-cache\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:13:49.321303 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.320683 5805 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-tmp-dir\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:13:49.322652 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.322482 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-home" pod "43eecb04-1091-4dde-9b9c-42a0fb28f08f" (UID: "43eecb04-1091-4dde-9b9c-42a0fb28f08f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:49.324772 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.324092 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-dshm" pod "43eecb04-1091-4dde-9b9c-42a0fb28f08f" (UID: "43eecb04-1091-4dde-9b9c-42a0fb28f08f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:49.324772 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.324711 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-dshm" pod "2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" (UID: "2353fc1e-3b17-41f1-a8f4-47c2bc2cc260"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:49.324772 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.324742 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-home" pod "2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" (UID: "2353fc1e-3b17-41f1-a8f4-47c2bc2cc260"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:49.325375 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.325340 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-tmp-dir" pod "43eecb04-1091-4dde-9b9c-42a0fb28f08f" (UID: "43eecb04-1091-4dde-9b9c-42a0fb28f08f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:49.325563 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.325531 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-model-cache" pod "43eecb04-1091-4dde-9b9c-42a0fb28f08f" (UID: "43eecb04-1091-4dde-9b9c-42a0fb28f08f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:49.326802 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.326772 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-kube-api-access-nlfm2" pod "2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" (UID: "2353fc1e-3b17-41f1-a8f4-47c2bc2cc260"). InnerVolumeSpecName "kube-api-access-nlfm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:13:49.327918 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.327882 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43eecb04-1091-4dde-9b9c-42a0fb28f08f-tls-certs" pod "43eecb04-1091-4dde-9b9c-42a0fb28f08f" (UID: "43eecb04-1091-4dde-9b9c-42a0fb28f08f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:13:49.328348 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.328323 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43eecb04-1091-4dde-9b9c-42a0fb28f08f-kube-api-access-jj2pv" pod "43eecb04-1091-4dde-9b9c-42a0fb28f08f" (UID: "43eecb04-1091-4dde-9b9c-42a0fb28f08f"). InnerVolumeSpecName "kube-api-access-jj2pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:13:49.328630 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.328605 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-tls-certs" pod "2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" (UID: "2353fc1e-3b17-41f1-a8f4-47c2bc2cc260"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:13:49.415592 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.415458 5805 generic.go:356] "Generic (PLEG): container finished" podID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerID="5e603a64e0fa63996c250c63a20f8e96e690b8ffa7220370c0cea00a274492fe" exitCode=0 Nov 03 18:13:49.415592 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.415515 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" Nov 03 18:13:49.415592 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.415564 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" event={"ID":"43eecb04-1091-4dde-9b9c-42a0fb28f08f","Type":"ContainerDied","Data":"5e603a64e0fa63996c250c63a20f8e96e690b8ffa7220370c0cea00a274492fe"} Nov 03 18:13:49.416327 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.415774 5805 scope.go:122] "RemoveContainer" containerID="5e603a64e0fa63996c250c63a20f8e96e690b8ffa7220370c0cea00a274492fe" Nov 03 18:13:49.416327 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.415701 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8" event={"ID":"43eecb04-1091-4dde-9b9c-42a0fb28f08f","Type":"ContainerDied","Data":"ed0b4fd4b2fb194ea4282eda5de9a76bded8f8cbb57d48fd961d4096218bbd7e"} Nov 03 18:13:49.418558 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.418532 5805 generic.go:356] "Generic (PLEG): container finished" podID="2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" containerID="b639272e70bb94c33b737d2f51c99472cd6928386466bd3dd69ddbc557ecbc66" exitCode=0 Nov 03 18:13:49.418798 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.418779 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" event={"ID":"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260","Type":"ContainerDied","Data":"b639272e70bb94c33b737d2f51c99472cd6928386466bd3dd69ddbc557ecbc66"} Nov 03 18:13:49.418873 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.418812 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" event={"ID":"2353fc1e-3b17-41f1-a8f4-47c2bc2cc260","Type":"ContainerDied","Data":"c6011e3c9888333a7e6062386807e928036b75d07b7a0545200af1d650f9c0eb"} Nov 03 18:13:49.418873 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.418868 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52" Nov 03 18:13:49.426265 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.426244 5805 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-dshm\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:13:49.426380 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.426269 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlfm2\" (UniqueName: \"kubernetes.io/projected/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-kube-api-access-nlfm2\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:13:49.426380 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.426284 5805 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-home\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:13:49.426380 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.426296 5805 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-tls-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:13:49.426380 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.426320 5805 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260-dshm\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:13:49.426380 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.426341 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj2pv\" (UniqueName: \"kubernetes.io/projected/43eecb04-1091-4dde-9b9c-42a0fb28f08f-kube-api-access-jj2pv\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:13:49.426380 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.426355 5805 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43eecb04-1091-4dde-9b9c-42a0fb28f08f-tls-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:13:49.426380 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.426369 5805 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-tmp-dir\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:13:49.426757 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.426382 5805 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-model-cache\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:13:49.426757 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.426412 5805 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43eecb04-1091-4dde-9b9c-42a0fb28f08f-home\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:13:49.433176 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.433157 5805 scope.go:122] "RemoveContainer" containerID="5e603a64e0fa63996c250c63a20f8e96e690b8ffa7220370c0cea00a274492fe" Nov 03 18:13:49.433741 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:13:49.433479 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e603a64e0fa63996c250c63a20f8e96e690b8ffa7220370c0cea00a274492fe\": container with ID starting with 5e603a64e0fa63996c250c63a20f8e96e690b8ffa7220370c0cea00a274492fe not found: ID does not exist" containerID="5e603a64e0fa63996c250c63a20f8e96e690b8ffa7220370c0cea00a274492fe" Nov 03 18:13:49.433741 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.433505 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e603a64e0fa63996c250c63a20f8e96e690b8ffa7220370c0cea00a274492fe"} err="failed to get container status \"5e603a64e0fa63996c250c63a20f8e96e690b8ffa7220370c0cea00a274492fe\": rpc error: code = NotFound desc = could not find container \"5e603a64e0fa63996c250c63a20f8e96e690b8ffa7220370c0cea00a274492fe\": container with ID starting with 5e603a64e0fa63996c250c63a20f8e96e690b8ffa7220370c0cea00a274492fe not found: ID does not exist" Nov 03 18:13:49.433741 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.433522 5805 scope.go:122] "RemoveContainer" containerID="b639272e70bb94c33b737d2f51c99472cd6928386466bd3dd69ddbc557ecbc66" Nov 03 18:13:49.443286 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.443268 5805 scope.go:122] "RemoveContainer" containerID="b639272e70bb94c33b737d2f51c99472cd6928386466bd3dd69ddbc557ecbc66" Nov 03 18:13:49.443557 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:13:49.443535 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b639272e70bb94c33b737d2f51c99472cd6928386466bd3dd69ddbc557ecbc66\": container with ID starting with b639272e70bb94c33b737d2f51c99472cd6928386466bd3dd69ddbc557ecbc66 not found: ID does not exist" containerID="b639272e70bb94c33b737d2f51c99472cd6928386466bd3dd69ddbc557ecbc66" Nov 03 18:13:49.443606 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.443566 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b639272e70bb94c33b737d2f51c99472cd6928386466bd3dd69ddbc557ecbc66"} err="failed to get container status \"b639272e70bb94c33b737d2f51c99472cd6928386466bd3dd69ddbc557ecbc66\": rpc error: code = NotFound desc = could not find container \"b639272e70bb94c33b737d2f51c99472cd6928386466bd3dd69ddbc557ecbc66\": container with ID starting with b639272e70bb94c33b737d2f51c99472cd6928386466bd3dd69ddbc557ecbc66 not found: ID does not exist" Nov 03 18:13:49.595215 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.595155 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8"] Nov 03 18:13:49.598698 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.598674 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0ccbb9/premium-w0-0ccbb9-kserve-dcfc69489-kczk8"] Nov 03 18:13:49.700010 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.699969 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52"] Nov 03 18:13:49.703002 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:49.702977 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-0ccbb9/distinct-w0-0ccbb9-kserve-5589bf4d88-82b52"] Nov 03 18:13:50.981498 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:50.981467 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" path="/var/lib/kubelet/pods/2353fc1e-3b17-41f1-a8f4-47c2bc2cc260/volumes" Nov 03 18:13:50.982031 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:50.982016 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" path="/var/lib/kubelet/pods/43eecb04-1091-4dde-9b9c-42a0fb28f08f/volumes" Nov 03 18:13:51.725080 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:51.725036 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:52.664631 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:52.664591 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:52.871458 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:52.871420 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:53.492621 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:53.492590 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:53.700249 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:53.700217 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:53.903419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:53.903362 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:54.381845 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:54.381808 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:55.197387 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:55.197346 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:55.417820 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:55.417787 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:55.642414 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:55.642174 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:55.929798 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:55.929719 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:56.128084 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:56.128049 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:56.254769 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:13:56.254630 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:13:56.460672 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:56.460631 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:13:58.420854 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:13:58.420815 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:00.365604 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:00.365568 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz"] Nov 03 18:14:00.366079 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:00.365829 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" podUID="984bc848-d32b-43d0-92cb-702cb3fa8f87" containerName="istio-proxy" containerID="cri-o://9830bd97d1bf5cd80ed4a807152b8bf20ead7df7afe8deb2e8034f1d0cf08301" gracePeriod=30 Nov 03 18:14:00.379270 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:00.379243 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:00.582070 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:00.582033 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:00.796499 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:00.793909 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:01.364451 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:01.364414 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm"] Nov 03 18:14:01.410806 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:01.410771 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r"] Nov 03 18:14:01.579631 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:01.579598 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:03.433528 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:03.432006 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:04.861174 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:04.861136 5805 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:14:04.861592 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:04.861216 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" podUID="984bc848-d32b-43d0-92cb-702cb3fa8f87" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:14:05.539887 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.539857 5805 generic.go:356] "Generic (PLEG): container finished" podID="984bc848-d32b-43d0-92cb-702cb3fa8f87" containerID="9830bd97d1bf5cd80ed4a807152b8bf20ead7df7afe8deb2e8034f1d0cf08301" exitCode=0 Nov 03 18:14:05.540045 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.539930 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" event={"ID":"984bc848-d32b-43d0-92cb-702cb3fa8f87","Type":"ContainerDied","Data":"9830bd97d1bf5cd80ed4a807152b8bf20ead7df7afe8deb2e8034f1d0cf08301"} Nov 03 18:14:05.647938 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.647878 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:14:05.713591 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.713555 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-credential-socket\") pod \"984bc848-d32b-43d0-92cb-702cb3fa8f87\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " Nov 03 18:14:05.713784 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.713637 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-workload-certs\") pod \"984bc848-d32b-43d0-92cb-702cb3fa8f87\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " Nov 03 18:14:05.713784 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.713664 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-token\" (UniqueName: \"kubernetes.io/projected/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-token\") pod \"984bc848-d32b-43d0-92cb-702cb3fa8f87\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " Nov 03 18:14:05.713784 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.713701 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-podinfo\") pod \"984bc848-d32b-43d0-92cb-702cb3fa8f87\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " Nov 03 18:14:05.713784 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.713754 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-envoy\") pod \"984bc848-d32b-43d0-92cb-702cb3fa8f87\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " Nov 03 18:14:05.714000 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.713796 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-data\") pod \"984bc848-d32b-43d0-92cb-702cb3fa8f87\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " Nov 03 18:14:05.714000 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.713895 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-credential-socket" pod "984bc848-d32b-43d0-92cb-702cb3fa8f87" (UID: "984bc848-d32b-43d0-92cb-702cb3fa8f87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:05.714000 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.713916 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/984bc848-d32b-43d0-92cb-702cb3fa8f87-kube-api-access-fz8bb\" (UniqueName: \"kubernetes.io/projected/984bc848-d32b-43d0-92cb-702cb3fa8f87-kube-api-access-fz8bb\") pod \"984bc848-d32b-43d0-92cb-702cb3fa8f87\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " Nov 03 18:14:05.714000 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.713980 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-workload-certs" pod "984bc848-d32b-43d0-92cb-702cb3fa8f87" (UID: "984bc848-d32b-43d0-92cb-702cb3fa8f87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:05.714000 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.713991 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-workload-socket\") pod \"984bc848-d32b-43d0-92cb-702cb3fa8f87\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " Nov 03 18:14:05.714276 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.714042 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/984bc848-d32b-43d0-92cb-702cb3fa8f87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984bc848-d32b-43d0-92cb-702cb3fa8f87-istiod-ca-cert\") pod \"984bc848-d32b-43d0-92cb-702cb3fa8f87\" (UID: \"984bc848-d32b-43d0-92cb-702cb3fa8f87\") " Nov 03 18:14:05.714276 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.714123 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-data" pod "984bc848-d32b-43d0-92cb-702cb3fa8f87" (UID: "984bc848-d32b-43d0-92cb-702cb3fa8f87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:05.714276 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.714244 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-workload-socket" pod "984bc848-d32b-43d0-92cb-702cb3fa8f87" (UID: "984bc848-d32b-43d0-92cb-702cb3fa8f87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:05.714557 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.714532 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/984bc848-d32b-43d0-92cb-702cb3fa8f87-istiod-ca-cert" pod "984bc848-d32b-43d0-92cb-702cb3fa8f87" (UID: "984bc848-d32b-43d0-92cb-702cb3fa8f87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:14:05.714845 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.714821 5805 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-credential-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:05.714971 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.714848 5805 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-workload-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:05.714971 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.714865 5805 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-data\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:05.714971 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.714881 5805 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-workload-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:05.714971 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.714895 5805 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/984bc848-d32b-43d0-92cb-702cb3fa8f87-istiod-ca-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:05.716079 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.716051 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-podinfo" pod "984bc848-d32b-43d0-92cb-702cb3fa8f87" (UID: "984bc848-d32b-43d0-92cb-702cb3fa8f87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:14:05.716174 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.716150 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984bc848-d32b-43d0-92cb-702cb3fa8f87-kube-api-access-fz8bb" pod "984bc848-d32b-43d0-92cb-702cb3fa8f87" (UID: "984bc848-d32b-43d0-92cb-702cb3fa8f87"). InnerVolumeSpecName "kube-api-access-fz8bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:14:05.716223 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.716174 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-envoy" pod "984bc848-d32b-43d0-92cb-702cb3fa8f87" (UID: "984bc848-d32b-43d0-92cb-702cb3fa8f87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:05.716223 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.716176 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-token" pod "984bc848-d32b-43d0-92cb-702cb3fa8f87" (UID: "984bc848-d32b-43d0-92cb-702cb3fa8f87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:14:05.817930 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.817888 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:05.817930 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.817928 5805 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-podinfo\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:05.818119 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.817940 5805 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/984bc848-d32b-43d0-92cb-702cb3fa8f87-istio-envoy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:05.818119 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:05.817949 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz8bb\" (UniqueName: \"kubernetes.io/projected/984bc848-d32b-43d0-92cb-702cb3fa8f87-kube-api-access-fz8bb\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:06.548897 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:06.548871 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" Nov 03 18:14:06.549356 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:06.548860 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz" event={"ID":"984bc848-d32b-43d0-92cb-702cb3fa8f87","Type":"ContainerDied","Data":"504ec44e7cebd0aebf0237b4fc97a038a2af21e3b946fb998a394614b6de97cd"} Nov 03 18:14:06.549356 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:06.549016 5805 scope.go:122] "RemoveContainer" containerID="9830bd97d1bf5cd80ed4a807152b8bf20ead7df7afe8deb2e8034f1d0cf08301" Nov 03 18:14:06.758720 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:06.758683 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz"] Nov 03 18:14:06.774585 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:06.769011 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-2fqmz"] Nov 03 18:14:06.776456 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:06.776426 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:06.978357 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:06.978319 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="984bc848-d32b-43d0-92cb-702cb3fa8f87" path="/var/lib/kubelet/pods/984bc848-d32b-43d0-92cb-702cb3fa8f87/volumes" Nov 03 18:14:08.363692 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:08.363647 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:11.419059 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:11.419017 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:11.881073 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:11.881041 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:12.082366 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:12.082324 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:12.398253 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:12.398218 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:12.603855 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:12.603822 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:12.985281 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:12.985247 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:13.396262 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:13.396224 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:13.628655 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:13.628621 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:13.846581 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:13.846541 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:14.155473 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:14.155344 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:14.740483 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:14.740445 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:16.386701 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:16.386655 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerName="main" containerID="cri-o://eb331f18634d0b57bf88d0a5527ae4e047220de90cc067ba5e050bfcf0537112" gracePeriod=45 Nov 03 18:14:16.452526 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:16.452483 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" containerID="cri-o://ed963466efb705d90a5ba7358261a0b54e4a22282ea43b01eccc2c36a9a648e4" gracePeriod=45 Nov 03 18:14:16.505645 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:16.505602 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": dial tcp 10.134.0.66:8000: connect: connection refused" Nov 03 18:14:16.552939 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:16.552898 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:16.669142 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:16.669111 5805 generic.go:356] "Generic (PLEG): container finished" podID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerID="eb331f18634d0b57bf88d0a5527ae4e047220de90cc067ba5e050bfcf0537112" exitCode=0 Nov 03 18:14:16.669362 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:16.669340 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" event={"ID":"a124942f-4351-4a09-8506-bf2bda9f71a3","Type":"ContainerDied","Data":"eb331f18634d0b57bf88d0a5527ae4e047220de90cc067ba5e050bfcf0537112"} Nov 03 18:14:16.673530 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:16.673378 5805 generic.go:356] "Generic (PLEG): container finished" podID="c968ad12-4333-4704-94f9-e4d34b204c81" containerID="ed963466efb705d90a5ba7358261a0b54e4a22282ea43b01eccc2c36a9a648e4" exitCode=0 Nov 03 18:14:16.673530 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:16.673448 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" event={"ID":"c968ad12-4333-4704-94f9-e4d34b204c81","Type":"ContainerDied","Data":"ed963466efb705d90a5ba7358261a0b54e4a22282ea43b01eccc2c36a9a648e4"} Nov 03 18:14:16.768603 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:16.768139 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:16.876476 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:16.875956 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:14:16.880719 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:16.880691 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 18:14:17.026758 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.026663 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-home\" (UniqueName: \"kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-home\") pod \"c968ad12-4333-4704-94f9-e4d34b204c81\" (UID: \"c968ad12-4333-4704-94f9-e4d34b204c81\") " Nov 03 18:14:17.026758 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.026743 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a124942f-4351-4a09-8506-bf2bda9f71a3-kube-api-access-dsf6r\" (UniqueName: \"kubernetes.io/projected/a124942f-4351-4a09-8506-bf2bda9f71a3-kube-api-access-dsf6r\") pod \"a124942f-4351-4a09-8506-bf2bda9f71a3\" (UID: \"a124942f-4351-4a09-8506-bf2bda9f71a3\") " Nov 03 18:14:17.027087 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.026775 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-model-cache\") pod \"c968ad12-4333-4704-94f9-e4d34b204c81\" (UID: \"c968ad12-4333-4704-94f9-e4d34b204c81\") " Nov 03 18:14:17.027087 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.026837 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-home\" (UniqueName: \"kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-home\") pod \"a124942f-4351-4a09-8506-bf2bda9f71a3\" (UID: \"a124942f-4351-4a09-8506-bf2bda9f71a3\") " Nov 03 18:14:17.027087 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.026875 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c968ad12-4333-4704-94f9-e4d34b204c81-tls-certs\" (UniqueName: \"kubernetes.io/secret/c968ad12-4333-4704-94f9-e4d34b204c81-tls-certs\") pod \"c968ad12-4333-4704-94f9-e4d34b204c81\" (UID: \"c968ad12-4333-4704-94f9-e4d34b204c81\") " Nov 03 18:14:17.027087 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.027050 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-dshm\") pod \"c968ad12-4333-4704-94f9-e4d34b204c81\" (UID: \"c968ad12-4333-4704-94f9-e4d34b204c81\") " Nov 03 18:14:17.027087 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.027054 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-home" pod "c968ad12-4333-4704-94f9-e4d34b204c81" (UID: "c968ad12-4333-4704-94f9-e4d34b204c81"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:17.027347 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.027107 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-tmp-dir\") pod \"c968ad12-4333-4704-94f9-e4d34b204c81\" (UID: \"c968ad12-4333-4704-94f9-e4d34b204c81\") " Nov 03 18:14:17.027347 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.027148 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-tmp-dir\") pod \"a124942f-4351-4a09-8506-bf2bda9f71a3\" (UID: \"a124942f-4351-4a09-8506-bf2bda9f71a3\") " Nov 03 18:14:17.027347 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.027185 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a124942f-4351-4a09-8506-bf2bda9f71a3-tls-certs\" (UniqueName: \"kubernetes.io/secret/a124942f-4351-4a09-8506-bf2bda9f71a3-tls-certs\") pod \"a124942f-4351-4a09-8506-bf2bda9f71a3\" (UID: \"a124942f-4351-4a09-8506-bf2bda9f71a3\") " Nov 03 18:14:17.027347 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.027216 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c968ad12-4333-4704-94f9-e4d34b204c81-kube-api-access-tw6sp\" (UniqueName: \"kubernetes.io/projected/c968ad12-4333-4704-94f9-e4d34b204c81-kube-api-access-tw6sp\") pod \"c968ad12-4333-4704-94f9-e4d34b204c81\" (UID: \"c968ad12-4333-4704-94f9-e4d34b204c81\") " Nov 03 18:14:17.027347 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.027252 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-dshm\") pod \"a124942f-4351-4a09-8506-bf2bda9f71a3\" (UID: \"a124942f-4351-4a09-8506-bf2bda9f71a3\") " Nov 03 18:14:17.027347 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.027279 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-model-cache\") pod \"a124942f-4351-4a09-8506-bf2bda9f71a3\" (UID: \"a124942f-4351-4a09-8506-bf2bda9f71a3\") " Nov 03 18:14:17.027347 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.027290 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-home" pod "a124942f-4351-4a09-8506-bf2bda9f71a3" (UID: "a124942f-4351-4a09-8506-bf2bda9f71a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:17.029997 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.028026 5805 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-home\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:17.029997 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.028049 5805 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-home\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:17.029997 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.028255 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-model-cache" pod "c968ad12-4333-4704-94f9-e4d34b204c81" (UID: "c968ad12-4333-4704-94f9-e4d34b204c81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:17.029997 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.029182 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-tmp-dir" pod "c968ad12-4333-4704-94f9-e4d34b204c81" (UID: "c968ad12-4333-4704-94f9-e4d34b204c81"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:17.029997 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.029662 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-tmp-dir" pod "a124942f-4351-4a09-8506-bf2bda9f71a3" (UID: "a124942f-4351-4a09-8506-bf2bda9f71a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:17.029997 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.029967 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-model-cache" pod "a124942f-4351-4a09-8506-bf2bda9f71a3" (UID: "a124942f-4351-4a09-8506-bf2bda9f71a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:17.031779 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.031617 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c968ad12-4333-4704-94f9-e4d34b204c81-tls-certs" pod "c968ad12-4333-4704-94f9-e4d34b204c81" (UID: "c968ad12-4333-4704-94f9-e4d34b204c81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:14:17.032069 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.032044 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c968ad12-4333-4704-94f9-e4d34b204c81-kube-api-access-tw6sp" pod "c968ad12-4333-4704-94f9-e4d34b204c81" (UID: "c968ad12-4333-4704-94f9-e4d34b204c81"). InnerVolumeSpecName "kube-api-access-tw6sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:14:17.032292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.032273 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a124942f-4351-4a09-8506-bf2bda9f71a3-tls-certs" pod "a124942f-4351-4a09-8506-bf2bda9f71a3" (UID: "a124942f-4351-4a09-8506-bf2bda9f71a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:14:17.035144 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.035116 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-dshm" pod "c968ad12-4333-4704-94f9-e4d34b204c81" (UID: "c968ad12-4333-4704-94f9-e4d34b204c81"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:17.035144 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.035131 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-dshm" pod "a124942f-4351-4a09-8506-bf2bda9f71a3" (UID: "a124942f-4351-4a09-8506-bf2bda9f71a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:17.036578 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.036550 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a124942f-4351-4a09-8506-bf2bda9f71a3-kube-api-access-dsf6r" pod "a124942f-4351-4a09-8506-bf2bda9f71a3" (UID: "a124942f-4351-4a09-8506-bf2bda9f71a3"). InnerVolumeSpecName "kube-api-access-dsf6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:14:17.130022 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.129978 5805 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-dshm\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:17.130022 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.130021 5805 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-tmp-dir\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:17.130250 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.130037 5805 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-tmp-dir\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:17.130250 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.130050 5805 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a124942f-4351-4a09-8506-bf2bda9f71a3-tls-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:17.130250 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.130065 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw6sp\" (UniqueName: \"kubernetes.io/projected/c968ad12-4333-4704-94f9-e4d34b204c81-kube-api-access-tw6sp\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:17.130250 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.130081 5805 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-dshm\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:17.130250 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.130095 5805 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a124942f-4351-4a09-8506-bf2bda9f71a3-model-cache\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:17.130250 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.130110 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsf6r\" (UniqueName: \"kubernetes.io/projected/a124942f-4351-4a09-8506-bf2bda9f71a3-kube-api-access-dsf6r\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:17.130250 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.130123 5805 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c968ad12-4333-4704-94f9-e4d34b204c81-model-cache\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:17.130250 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.130136 5805 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c968ad12-4333-4704-94f9-e4d34b204c81-tls-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:17.754387 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.754281 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" event={"ID":"c968ad12-4333-4704-94f9-e4d34b204c81","Type":"ContainerDied","Data":"422e70a148ccff2876b9943e197cf2ccfc1cb943416363a256e431494467b00a"} Nov 03 18:14:17.759138 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.759106 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" event={"ID":"a124942f-4351-4a09-8506-bf2bda9f71a3","Type":"ContainerDied","Data":"9ea5d208830ad4f1fa6449a87646145d57825f36b9a36d8c6eb03b276a2cd0bb"} Nov 03 18:14:17.759416 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.758591 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm" Nov 03 18:14:17.772455 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.767519 5805 scope.go:122] "RemoveContainer" containerID="ed963466efb705d90a5ba7358261a0b54e4a22282ea43b01eccc2c36a9a648e4" Nov 03 18:14:17.772455 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.756599 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" Nov 03 18:14:17.814313 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.812825 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:14:17.835591 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.835556 5805 scope.go:122] "RemoveContainer" containerID="eb331f18634d0b57bf88d0a5527ae4e047220de90cc067ba5e050bfcf0537112" Nov 03 18:14:17.983461 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.983424 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm"] Nov 03 18:14:17.987144 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:17.987118 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-0adcbc/distinct-2-w3-0adcbc-kserve-796bdc5776-85dgm"] Nov 03 18:14:18.091101 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:18.091054 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r"] Nov 03 18:14:18.109463 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:18.105642 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-0adcbc/simulator-w3-0adcbc-kserve-77d9b6c945-sh69r"] Nov 03 18:14:18.322028 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:18.321993 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:18.540093 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:18.540001 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:18.743347 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:18.743301 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:18.891769 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:18.891732 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg"] Nov 03 18:14:19.003890 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:19.000688 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" path="/var/lib/kubelet/pods/a124942f-4351-4a09-8506-bf2bda9f71a3/volumes" Nov 03 18:14:19.003890 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:19.001367 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" path="/var/lib/kubelet/pods/c968ad12-4333-4704-94f9-e4d34b204c81/volumes" Nov 03 18:14:19.361906 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:19.361852 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:19.567187 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:19.567154 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:19.783613 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:19.783528 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:23.325719 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:23.325657 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:23.590762 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:23.590668 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:23.809858 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:23.809825 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:23.892741 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:23.892656 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg" podUID="7195e16b-0603-4421-92be-3cff9a0fa7eb" containerName="payload-processing" containerID="cri-o://a695737ae7791cfd82401e824525e5204470e0e736496fe78d02bbca854f76db" gracePeriod=25 Nov 03 18:14:26.236715 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:14:26.236677 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:14:27.998273 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:27.998229 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:29.974624 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:29.974587 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg" podUID="7195e16b-0603-4421-92be-3cff9a0fa7eb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:14:32.431445 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:32.431372 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:36.968480 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:36.968124 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:38.688670 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:38.688446 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:38.902801 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:38.902769 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:39.482127 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:39.482094 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:39.975186 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:39.975151 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg" podUID="7195e16b-0603-4421-92be-3cff9a0fa7eb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:14:41.547067 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:41.546137 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:41.744376 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:41.744336 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:44.266585 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:44.266560 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg" Nov 03 18:14:44.369891 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:44.369856 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7195e16b-0603-4421-92be-3cff9a0fa7eb-kube-api-access-d9n7m\" (UniqueName: \"kubernetes.io/projected/7195e16b-0603-4421-92be-3cff9a0fa7eb-kube-api-access-d9n7m\") pod \"7195e16b-0603-4421-92be-3cff9a0fa7eb\" (UID: \"7195e16b-0603-4421-92be-3cff9a0fa7eb\") " Nov 03 18:14:44.370038 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:44.370016 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7195e16b-0603-4421-92be-3cff9a0fa7eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7195e16b-0603-4421-92be-3cff9a0fa7eb-plugins-config-volume\") pod \"7195e16b-0603-4421-92be-3cff9a0fa7eb\" (UID: \"7195e16b-0603-4421-92be-3cff9a0fa7eb\") " Nov 03 18:14:44.370544 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:44.370510 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7195e16b-0603-4421-92be-3cff9a0fa7eb-plugins-config-volume" pod "7195e16b-0603-4421-92be-3cff9a0fa7eb" (UID: "7195e16b-0603-4421-92be-3cff9a0fa7eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:14:44.370660 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:44.370646 5805 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7195e16b-0603-4421-92be-3cff9a0fa7eb-plugins-config-volume\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:44.372069 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:44.372043 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7195e16b-0603-4421-92be-3cff9a0fa7eb-kube-api-access-d9n7m" pod "7195e16b-0603-4421-92be-3cff9a0fa7eb" (UID: "7195e16b-0603-4421-92be-3cff9a0fa7eb"). InnerVolumeSpecName "kube-api-access-d9n7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:14:44.471525 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:44.471491 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9n7m\" (UniqueName: \"kubernetes.io/projected/7195e16b-0603-4421-92be-3cff9a0fa7eb-kube-api-access-d9n7m\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:45.011557 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:45.011526 5805 generic.go:356] "Generic (PLEG): container finished" podID="7195e16b-0603-4421-92be-3cff9a0fa7eb" containerID="a695737ae7791cfd82401e824525e5204470e0e736496fe78d02bbca854f76db" exitCode=0 Nov 03 18:14:45.011709 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:45.011562 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg" event={"ID":"7195e16b-0603-4421-92be-3cff9a0fa7eb","Type":"ContainerDied","Data":"a695737ae7791cfd82401e824525e5204470e0e736496fe78d02bbca854f76db"} Nov 03 18:14:45.011709 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:45.011614 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg" event={"ID":"7195e16b-0603-4421-92be-3cff9a0fa7eb","Type":"ContainerDied","Data":"6f83557bd8502bbcb33a06db49e0a4cb134f3e362eac2c44331d6c8c01f73ec0"} Nov 03 18:14:45.011709 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:45.011635 5805 scope.go:122] "RemoveContainer" containerID="a695737ae7791cfd82401e824525e5204470e0e736496fe78d02bbca854f76db" Nov 03 18:14:45.011709 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:45.011573 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg" Nov 03 18:14:45.021165 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:45.021142 5805 scope.go:122] "RemoveContainer" containerID="a695737ae7791cfd82401e824525e5204470e0e736496fe78d02bbca854f76db" Nov 03 18:14:45.021435 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:14:45.021414 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a695737ae7791cfd82401e824525e5204470e0e736496fe78d02bbca854f76db\": container with ID starting with a695737ae7791cfd82401e824525e5204470e0e736496fe78d02bbca854f76db not found: ID does not exist" containerID="a695737ae7791cfd82401e824525e5204470e0e736496fe78d02bbca854f76db" Nov 03 18:14:45.021504 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:45.021441 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a695737ae7791cfd82401e824525e5204470e0e736496fe78d02bbca854f76db"} err="failed to get container status \"a695737ae7791cfd82401e824525e5204470e0e736496fe78d02bbca854f76db\": rpc error: code = NotFound desc = could not find container \"a695737ae7791cfd82401e824525e5204470e0e736496fe78d02bbca854f76db\": container with ID starting with a695737ae7791cfd82401e824525e5204470e0e736496fe78d02bbca854f76db not found: ID does not exist" Nov 03 18:14:45.166829 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:45.166799 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg"] Nov 03 18:14:45.212755 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:45.212720 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-aab753-cf998fd9-5r6fg"] Nov 03 18:14:45.324835 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:45.324803 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:45.497523 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:45.497485 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74"] Nov 03 18:14:45.497707 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:45.497686 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" podUID="0ac493e2-b140-4b42-93ca-fba0da0079de" containerName="maas-api" containerID="cri-o://ea3b995359065459a8d769075f688fdb27ef865e5fd0dbdac8380706af36308d" gracePeriod=30 Nov 03 18:14:45.663445 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:45.663260 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:45.942639 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:45.942564 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" Nov 03 18:14:46.110730 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:46.110688 5805 generic.go:356] "Generic (PLEG): container finished" podID="0ac493e2-b140-4b42-93ca-fba0da0079de" containerID="ea3b995359065459a8d769075f688fdb27ef865e5fd0dbdac8380706af36308d" exitCode=0 Nov 03 18:14:46.111287 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:46.111256 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" event={"ID":"0ac493e2-b140-4b42-93ca-fba0da0079de","Type":"ContainerDied","Data":"ea3b995359065459a8d769075f688fdb27ef865e5fd0dbdac8380706af36308d"} Nov 03 18:14:46.111553 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:46.111533 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" event={"ID":"0ac493e2-b140-4b42-93ca-fba0da0079de","Type":"ContainerDied","Data":"796ef35819e0821e0257249ae3c1087e0335590c53b61bab92dfc171f1d4e487"} Nov 03 18:14:46.111708 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:46.111694 5805 scope.go:122] "RemoveContainer" containerID="ea3b995359065459a8d769075f688fdb27ef865e5fd0dbdac8380706af36308d" Nov 03 18:14:46.111973 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:46.111957 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74" Nov 03 18:14:46.137207 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:46.132539 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ac493e2-b140-4b42-93ca-fba0da0079de-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac493e2-b140-4b42-93ca-fba0da0079de-openshift-service-ca\") pod \"0ac493e2-b140-4b42-93ca-fba0da0079de\" (UID: \"0ac493e2-b140-4b42-93ca-fba0da0079de\") " Nov 03 18:14:46.137207 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:46.132638 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ac493e2-b140-4b42-93ca-fba0da0079de-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac493e2-b140-4b42-93ca-fba0da0079de-metrics-certs\") pod \"0ac493e2-b140-4b42-93ca-fba0da0079de\" (UID: \"0ac493e2-b140-4b42-93ca-fba0da0079de\") " Nov 03 18:14:46.137207 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:46.132769 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ac493e2-b140-4b42-93ca-fba0da0079de-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac493e2-b140-4b42-93ca-fba0da0079de-maas-api-tls\") pod \"0ac493e2-b140-4b42-93ca-fba0da0079de\" (UID: \"0ac493e2-b140-4b42-93ca-fba0da0079de\") " Nov 03 18:14:46.137207 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:46.132888 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ac493e2-b140-4b42-93ca-fba0da0079de-kube-api-access-jxzmn\" (UniqueName: \"kubernetes.io/projected/0ac493e2-b140-4b42-93ca-fba0da0079de-kube-api-access-jxzmn\") pod \"0ac493e2-b140-4b42-93ca-fba0da0079de\" (UID: \"0ac493e2-b140-4b42-93ca-fba0da0079de\") " Nov 03 18:14:46.156768 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:46.156741 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac493e2-b140-4b42-93ca-fba0da0079de-openshift-service-ca" pod "0ac493e2-b140-4b42-93ca-fba0da0079de" (UID: "0ac493e2-b140-4b42-93ca-fba0da0079de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:14:46.157281 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:46.157249 5805 scope.go:122] "RemoveContainer" containerID="ea3b995359065459a8d769075f688fdb27ef865e5fd0dbdac8380706af36308d" Nov 03 18:14:46.158056 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:14:46.158028 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea3b995359065459a8d769075f688fdb27ef865e5fd0dbdac8380706af36308d\": container with ID starting with ea3b995359065459a8d769075f688fdb27ef865e5fd0dbdac8380706af36308d not found: ID does not exist" containerID="ea3b995359065459a8d769075f688fdb27ef865e5fd0dbdac8380706af36308d" Nov 03 18:14:46.158233 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:46.158206 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea3b995359065459a8d769075f688fdb27ef865e5fd0dbdac8380706af36308d"} err="failed to get container status \"ea3b995359065459a8d769075f688fdb27ef865e5fd0dbdac8380706af36308d\": rpc error: code = NotFound desc = could not find container \"ea3b995359065459a8d769075f688fdb27ef865e5fd0dbdac8380706af36308d\": container with ID starting with ea3b995359065459a8d769075f688fdb27ef865e5fd0dbdac8380706af36308d not found: ID does not exist" Nov 03 18:14:46.159615 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:46.159586 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ac493e2-b140-4b42-93ca-fba0da0079de-maas-api-tls" pod "0ac493e2-b140-4b42-93ca-fba0da0079de" (UID: "0ac493e2-b140-4b42-93ca-fba0da0079de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:14:46.174902 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:46.172444 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ac493e2-b140-4b42-93ca-fba0da0079de-metrics-certs" pod "0ac493e2-b140-4b42-93ca-fba0da0079de" (UID: "0ac493e2-b140-4b42-93ca-fba0da0079de"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:14:46.175331 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:46.175306 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac493e2-b140-4b42-93ca-fba0da0079de-kube-api-access-jxzmn" pod "0ac493e2-b140-4b42-93ca-fba0da0079de" (UID: "0ac493e2-b140-4b42-93ca-fba0da0079de"). InnerVolumeSpecName "kube-api-access-jxzmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:14:46.256971 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:46.256892 5805 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac493e2-b140-4b42-93ca-fba0da0079de-openshift-service-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:46.256971 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:46.256926 5805 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac493e2-b140-4b42-93ca-fba0da0079de-metrics-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:46.256971 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:46.256937 5805 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac493e2-b140-4b42-93ca-fba0da0079de-maas-api-tls\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:46.256971 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:46.256945 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxzmn\" (UniqueName: \"kubernetes.io/projected/0ac493e2-b140-4b42-93ca-fba0da0079de-kube-api-access-jxzmn\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:46.563872 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:46.557758 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74"] Nov 03 18:14:46.578475 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:46.574114 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-0ccbb9-648475764-wvq74"] Nov 03 18:14:46.977034 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:46.976955 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ac493e2-b140-4b42-93ca-fba0da0079de" path="/var/lib/kubelet/pods/0ac493e2-b140-4b42-93ca-fba0da0079de/volumes" Nov 03 18:14:46.977607 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:46.977581 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7195e16b-0603-4421-92be-3cff9a0fa7eb" path="/var/lib/kubelet/pods/7195e16b-0603-4421-92be-3cff9a0fa7eb/volumes" Nov 03 18:14:47.188038 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:47.188008 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:47.409154 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:47.409123 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:47.619285 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:47.619255 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:47.643889 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:47.643862 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26"] Nov 03 18:14:47.644087 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:47.644051 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" podUID="e1392256-ac5e-4f9a-90da-87b002ebd848" containerName="istio-proxy" containerID="cri-o://6d3a13a90e4fb354c9350b9ef157e101ede74be0cf2e0fc030532325f2bf02bf" gracePeriod=30 Nov 03 18:14:48.112747 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:48.112715 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:48.327440 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:48.319899 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:50.704781 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:50.704739 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:51.712797 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:51.712764 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:53.200956 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.200919 5805 generic.go:356] "Generic (PLEG): container finished" podID="e1392256-ac5e-4f9a-90da-87b002ebd848" containerID="6d3a13a90e4fb354c9350b9ef157e101ede74be0cf2e0fc030532325f2bf02bf" exitCode=0 Nov 03 18:14:53.201693 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.201642 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" event={"ID":"e1392256-ac5e-4f9a-90da-87b002ebd848","Type":"ContainerDied","Data":"6d3a13a90e4fb354c9350b9ef157e101ede74be0cf2e0fc030532325f2bf02bf"} Nov 03 18:14:53.410496 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.404971 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:53.513924 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.513897 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:14:53.649419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.634469 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:53.668471 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.657480 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1392256-ac5e-4f9a-90da-87b002ebd848-kube-api-access-9xvhz\" (UniqueName: \"kubernetes.io/projected/e1392256-ac5e-4f9a-90da-87b002ebd848-kube-api-access-9xvhz\") pod \"e1392256-ac5e-4f9a-90da-87b002ebd848\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " Nov 03 18:14:53.668471 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.657603 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-podinfo\") pod \"e1392256-ac5e-4f9a-90da-87b002ebd848\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " Nov 03 18:14:53.668471 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.657649 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-envoy\") pod \"e1392256-ac5e-4f9a-90da-87b002ebd848\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " Nov 03 18:14:53.668471 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.657706 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1392256-ac5e-4f9a-90da-87b002ebd848-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1392256-ac5e-4f9a-90da-87b002ebd848-istiod-ca-cert\") pod \"e1392256-ac5e-4f9a-90da-87b002ebd848\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " Nov 03 18:14:53.668471 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.657767 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-data\") pod \"e1392256-ac5e-4f9a-90da-87b002ebd848\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " Nov 03 18:14:53.668471 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.657817 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-workload-certs\") pod \"e1392256-ac5e-4f9a-90da-87b002ebd848\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " Nov 03 18:14:53.668471 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.657873 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-credential-socket\") pod \"e1392256-ac5e-4f9a-90da-87b002ebd848\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " Nov 03 18:14:53.668471 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.657963 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-workload-socket\") pod \"e1392256-ac5e-4f9a-90da-87b002ebd848\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " Nov 03 18:14:53.668471 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.658041 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-token\" (UniqueName: \"kubernetes.io/projected/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-token\") pod \"e1392256-ac5e-4f9a-90da-87b002ebd848\" (UID: \"e1392256-ac5e-4f9a-90da-87b002ebd848\") " Nov 03 18:14:53.668471 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.659462 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1392256-ac5e-4f9a-90da-87b002ebd848-istiod-ca-cert" pod "e1392256-ac5e-4f9a-90da-87b002ebd848" (UID: "e1392256-ac5e-4f9a-90da-87b002ebd848"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:14:53.668471 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.659937 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-workload-certs" pod "e1392256-ac5e-4f9a-90da-87b002ebd848" (UID: "e1392256-ac5e-4f9a-90da-87b002ebd848"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:53.668471 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.660040 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-data" pod "e1392256-ac5e-4f9a-90da-87b002ebd848" (UID: "e1392256-ac5e-4f9a-90da-87b002ebd848"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:53.668471 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.660055 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-credential-socket" pod "e1392256-ac5e-4f9a-90da-87b002ebd848" (UID: "e1392256-ac5e-4f9a-90da-87b002ebd848"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:53.668471 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.660067 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-workload-socket" pod "e1392256-ac5e-4f9a-90da-87b002ebd848" (UID: "e1392256-ac5e-4f9a-90da-87b002ebd848"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:53.668471 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.666085 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-envoy" pod "e1392256-ac5e-4f9a-90da-87b002ebd848" (UID: "e1392256-ac5e-4f9a-90da-87b002ebd848"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:53.678456 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.676574 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1392256-ac5e-4f9a-90da-87b002ebd848-kube-api-access-9xvhz" pod "e1392256-ac5e-4f9a-90da-87b002ebd848" (UID: "e1392256-ac5e-4f9a-90da-87b002ebd848"). InnerVolumeSpecName "kube-api-access-9xvhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:14:53.678456 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.676653 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-podinfo" pod "e1392256-ac5e-4f9a-90da-87b002ebd848" (UID: "e1392256-ac5e-4f9a-90da-87b002ebd848"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:14:53.691505 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.689022 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-token" pod "e1392256-ac5e-4f9a-90da-87b002ebd848" (UID: "e1392256-ac5e-4f9a-90da-87b002ebd848"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:14:53.760634 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.760600 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:53.760634 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.760631 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xvhz\" (UniqueName: \"kubernetes.io/projected/e1392256-ac5e-4f9a-90da-87b002ebd848-kube-api-access-9xvhz\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:53.760634 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.760641 5805 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-podinfo\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:53.760873 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.760650 5805 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-envoy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:53.760873 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.760659 5805 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1392256-ac5e-4f9a-90da-87b002ebd848-istiod-ca-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:53.760873 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.760667 5805 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-istio-data\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:53.760873 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.760676 5805 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-workload-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:53.760873 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.760686 5805 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-credential-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:53.760873 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.760694 5805 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1392256-ac5e-4f9a-90da-87b002ebd848-workload-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:14:53.848141 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:53.848106 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:54.209035 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:54.209009 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" Nov 03 18:14:54.209444 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:54.209003 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26" event={"ID":"e1392256-ac5e-4f9a-90da-87b002ebd848","Type":"ContainerDied","Data":"1303615f124d5bcef7c250aabc5779b027b9ba5a47782f13ccfcf2093442d74c"} Nov 03 18:14:54.209444 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:54.209138 5805 scope.go:122] "RemoveContainer" containerID="6d3a13a90e4fb354c9350b9ef157e101ede74be0cf2e0fc030532325f2bf02bf" Nov 03 18:14:54.297190 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:54.297163 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:54.413384 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:54.413350 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26"] Nov 03 18:14:54.459317 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:54.459251 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-86d26"] Nov 03 18:14:54.976976 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:54.976948 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1392256-ac5e-4f9a-90da-87b002ebd848" path="/var/lib/kubelet/pods/e1392256-ac5e-4f9a-90da-87b002ebd848/volumes" Nov 03 18:14:55.600436 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:55.596098 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:55.797165 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:55.797090 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:55.890606 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:55.890514 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4"] Nov 03 18:14:55.890776 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:55.890753 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" podUID="44286112-623b-4654-9b84-7eed4dd067d9" containerName="istio-proxy" containerID="cri-o://a0d08bd0b2d0ab1dc5283ad828636d0c9c8e079345aa0702b8234a2cb6485127" gracePeriod=30 Nov 03 18:14:55.900881 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:55.900843 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm"] Nov 03 18:14:55.901065 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:55.901044 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" podUID="348445a7-ec7a-454d-86aa-8ef0beeccb33" containerName="istio-proxy" containerID="cri-o://265c19c926adecfebfcbc5c7d777a8aab146a3624b3db170f0915a6532746351" gracePeriod=30 Nov 03 18:14:55.913471 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:55.913444 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw"] Nov 03 18:14:55.913652 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:55.913631 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" podUID="d5030e39-1c83-41e5-9484-f0b9c183084d" containerName="istio-proxy" containerID="cri-o://9f205abcc96a42eec8883c6dbb6cc8b779fa6d702a2750adf4bee81d89fd0c71" gracePeriod=30 Nov 03 18:14:56.010431 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:56.001623 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:56.240432 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:14:56.240318 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:14:56.993510 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:56.993475 5805 patch_prober.go:29] interesting pod/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:14:56.993903 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:56.993531 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" podUID="44286112-623b-4654-9b84-7eed4dd067d9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:14:57.793340 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:57.793305 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:14:58.488995 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:14:58.488962 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:00.170946 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.170908 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz"] Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.171792 5805 cpu_manager.go:414] "containerMap: removing container" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerName="main" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.171812 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerName="main" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.171827 5805 cpu_manager.go:414] "containerMap: removing container" podUID="aca6224c-f93a-4d51-b37e-f3368efa25dd" containerName="istio-proxy" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.171836 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aca6224c-f93a-4d51-b37e-f3368efa25dd" containerName="istio-proxy" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.171851 5805 cpu_manager.go:414] "containerMap: removing container" podUID="7195e16b-0603-4421-92be-3cff9a0fa7eb" containerName="payload-processing" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.171859 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7195e16b-0603-4421-92be-3cff9a0fa7eb" containerName="payload-processing" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.171879 5805 cpu_manager.go:414] "containerMap: removing container" podUID="2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" containerName="main" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.171888 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" containerName="main" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.171901 5805 cpu_manager.go:414] "containerMap: removing container" podUID="2df51a82-bd64-4ad8-a164-d5af9ee53e2d" containerName="istio-proxy" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.171909 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2df51a82-bd64-4ad8-a164-d5af9ee53e2d" containerName="istio-proxy" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.171929 5805 cpu_manager.go:414] "containerMap: removing container" podUID="86d8d1e1-cbb6-4e0f-93ac-9a19035e7681" containerName="istio-proxy" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.171937 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d8d1e1-cbb6-4e0f-93ac-9a19035e7681" containerName="istio-proxy" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.171947 5805 cpu_manager.go:414] "containerMap: removing container" podUID="e1392256-ac5e-4f9a-90da-87b002ebd848" containerName="istio-proxy" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.171955 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1392256-ac5e-4f9a-90da-87b002ebd848" containerName="istio-proxy" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.171980 5805 cpu_manager.go:414] "containerMap: removing container" podUID="984bc848-d32b-43d0-92cb-702cb3fa8f87" containerName="istio-proxy" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.171987 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="984bc848-d32b-43d0-92cb-702cb3fa8f87" containerName="istio-proxy" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.172001 5805 cpu_manager.go:414] "containerMap: removing container" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.172008 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.172024 5805 cpu_manager.go:414] "containerMap: removing container" podUID="0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" containerName="istio-proxy" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.172031 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" containerName="istio-proxy" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.172049 5805 cpu_manager.go:414] "containerMap: removing container" podUID="0ac493e2-b140-4b42-93ca-fba0da0079de" containerName="maas-api" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.172056 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ac493e2-b140-4b42-93ca-fba0da0079de" containerName="maas-api" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.172079 5805 cpu_manager.go:414] "containerMap: removing container" podUID="0b450130-5eea-4fe2-896c-62ba2fe84807" containerName="istio-proxy" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.172086 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b450130-5eea-4fe2-896c-62ba2fe84807" containerName="istio-proxy" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.172104 5805 cpu_manager.go:414] "containerMap: removing container" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.172111 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.172237 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b450130-5eea-4fe2-896c-62ba2fe84807" containerName="istio-proxy" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.172250 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="c968ad12-4333-4704-94f9-e4d34b204c81" containerName="main" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.172258 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="a124942f-4351-4a09-8506-bf2bda9f71a3" containerName="main" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.172264 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="aca6224c-f93a-4d51-b37e-f3368efa25dd" containerName="istio-proxy" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.172270 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="7195e16b-0603-4421-92be-3cff9a0fa7eb" containerName="payload-processing" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.172277 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="2353fc1e-3b17-41f1-a8f4-47c2bc2cc260" containerName="main" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.172284 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="2df51a82-bd64-4ad8-a164-d5af9ee53e2d" containerName="istio-proxy" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.172293 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d8d1e1-cbb6-4e0f-93ac-9a19035e7681" containerName="istio-proxy" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.172300 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1392256-ac5e-4f9a-90da-87b002ebd848" containerName="istio-proxy" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.172312 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="984bc848-d32b-43d0-92cb-702cb3fa8f87" containerName="istio-proxy" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.172321 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="43eecb04-1091-4dde-9b9c-42a0fb28f08f" containerName="main" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.172326 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="0133a1d6-2676-4f88-8fee-1dfe9c2b28ac" containerName="istio-proxy" Nov 03 18:15:00.173474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.172333 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ac493e2-b140-4b42-93ca-fba0da0079de" containerName="maas-api" Nov 03 18:15:00.175694 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.175677 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz" Nov 03 18:15:00.189072 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.189050 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg"] Nov 03 18:15:00.193387 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.193369 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf"] Nov 03 18:15:00.193519 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.193503 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg" Nov 03 18:15:00.197194 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.197176 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf" Nov 03 18:15:00.216989 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.216967 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp"] Nov 03 18:15:00.220749 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.220729 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v"] Nov 03 18:15:00.220889 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.220866 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp" Nov 03 18:15:00.224892 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.224875 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v" Nov 03 18:15:00.235054 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.235035 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\"" Nov 03 18:15:00.257105 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.257083 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7221deb2-4d2f-4f43-b7a5-064464fc3991-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz\" (UID: \"7221deb2-4d2f-4f43-b7a5-064464fc3991\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz" Nov 03 18:15:00.257276 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.257249 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qfbj\" (UniqueName: \"kubernetes.io/projected/7221deb2-4d2f-4f43-b7a5-064464fc3991-kube-api-access-9qfbj\") pod \"maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz\" (UID: \"7221deb2-4d2f-4f43-b7a5-064464fc3991\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz" Nov 03 18:15:00.264490 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.264462 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz"] Nov 03 18:15:00.314151 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.313814 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg"] Nov 03 18:15:00.330512 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.322937 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf"] Nov 03 18:15:00.330512 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.328832 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp"] Nov 03 18:15:00.358565 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.358534 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78a7ba2-1042-492c-b52c-dca73afd27d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v\" (UID: \"b78a7ba2-1042-492c-b52c-dca73afd27d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v" Nov 03 18:15:00.358727 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.358595 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73900785-3360-4c42-8a5d-8c0e258fc637-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg\" (UID: \"73900785-3360-4c42-8a5d-8c0e258fc637\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg" Nov 03 18:15:00.358727 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.358675 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22875d5-06bf-4abe-9961-9f5c9922eac0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf\" (UID: \"d22875d5-06bf-4abe-9961-9f5c9922eac0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf" Nov 03 18:15:00.358849 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.358819 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qfbj\" (UniqueName: \"kubernetes.io/projected/7221deb2-4d2f-4f43-b7a5-064464fc3991-kube-api-access-9qfbj\") pod \"maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz\" (UID: \"7221deb2-4d2f-4f43-b7a5-064464fc3991\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz" Nov 03 18:15:00.358988 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.358930 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4crbc\" (UniqueName: \"kubernetes.io/projected/60483b95-19d2-46c3-ad70-5d8b36ffb572-kube-api-access-4crbc\") pod \"maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp\" (UID: \"60483b95-19d2-46c3-ad70-5d8b36ffb572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp" Nov 03 18:15:00.359061 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.359027 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6v4s\" (UniqueName: \"kubernetes.io/projected/d22875d5-06bf-4abe-9961-9f5c9922eac0-kube-api-access-p6v4s\") pod \"maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf\" (UID: \"d22875d5-06bf-4abe-9961-9f5c9922eac0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf" Nov 03 18:15:00.359192 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.359063 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60483b95-19d2-46c3-ad70-5d8b36ffb572-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp\" (UID: \"60483b95-19d2-46c3-ad70-5d8b36ffb572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp" Nov 03 18:15:00.359192 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.359082 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqxzm\" (UniqueName: \"kubernetes.io/projected/b78a7ba2-1042-492c-b52c-dca73afd27d5-kube-api-access-tqxzm\") pod \"maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v\" (UID: \"b78a7ba2-1042-492c-b52c-dca73afd27d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v" Nov 03 18:15:00.359192 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.359128 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7221deb2-4d2f-4f43-b7a5-064464fc3991-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz\" (UID: \"7221deb2-4d2f-4f43-b7a5-064464fc3991\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz" Nov 03 18:15:00.359364 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.359243 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48jrd\" (UniqueName: \"kubernetes.io/projected/73900785-3360-4c42-8a5d-8c0e258fc637-kube-api-access-48jrd\") pod \"maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg\" (UID: \"73900785-3360-4c42-8a5d-8c0e258fc637\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg" Nov 03 18:15:00.359782 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.359762 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7221deb2-4d2f-4f43-b7a5-064464fc3991-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz\" (UID: \"7221deb2-4d2f-4f43-b7a5-064464fc3991\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz" Nov 03 18:15:00.381529 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.381505 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v"] Nov 03 18:15:00.416432 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.416366 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qfbj\" (UniqueName: \"kubernetes.io/projected/7221deb2-4d2f-4f43-b7a5-064464fc3991-kube-api-access-9qfbj\") pod \"maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz\" (UID: \"7221deb2-4d2f-4f43-b7a5-064464fc3991\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz" Nov 03 18:15:00.461198 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.461124 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73900785-3360-4c42-8a5d-8c0e258fc637-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg\" (UID: \"73900785-3360-4c42-8a5d-8c0e258fc637\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg" Nov 03 18:15:00.461198 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.461165 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22875d5-06bf-4abe-9961-9f5c9922eac0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf\" (UID: \"d22875d5-06bf-4abe-9961-9f5c9922eac0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf" Nov 03 18:15:00.461390 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.461313 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4crbc\" (UniqueName: \"kubernetes.io/projected/60483b95-19d2-46c3-ad70-5d8b36ffb572-kube-api-access-4crbc\") pod \"maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp\" (UID: \"60483b95-19d2-46c3-ad70-5d8b36ffb572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp" Nov 03 18:15:00.461390 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.461379 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6v4s\" (UniqueName: \"kubernetes.io/projected/d22875d5-06bf-4abe-9961-9f5c9922eac0-kube-api-access-p6v4s\") pod \"maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf\" (UID: \"d22875d5-06bf-4abe-9961-9f5c9922eac0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf" Nov 03 18:15:00.461557 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.461457 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60483b95-19d2-46c3-ad70-5d8b36ffb572-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp\" (UID: \"60483b95-19d2-46c3-ad70-5d8b36ffb572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp" Nov 03 18:15:00.461557 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.461485 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqxzm\" (UniqueName: \"kubernetes.io/projected/b78a7ba2-1042-492c-b52c-dca73afd27d5-kube-api-access-tqxzm\") pod \"maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v\" (UID: \"b78a7ba2-1042-492c-b52c-dca73afd27d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v" Nov 03 18:15:00.461660 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.461576 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48jrd\" (UniqueName: \"kubernetes.io/projected/73900785-3360-4c42-8a5d-8c0e258fc637-kube-api-access-48jrd\") pod \"maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg\" (UID: \"73900785-3360-4c42-8a5d-8c0e258fc637\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg" Nov 03 18:15:00.461722 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.461686 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78a7ba2-1042-492c-b52c-dca73afd27d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v\" (UID: \"b78a7ba2-1042-492c-b52c-dca73afd27d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v" Nov 03 18:15:00.461955 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.461934 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22875d5-06bf-4abe-9961-9f5c9922eac0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf\" (UID: \"d22875d5-06bf-4abe-9961-9f5c9922eac0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf" Nov 03 18:15:00.462008 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.461938 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73900785-3360-4c42-8a5d-8c0e258fc637-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg\" (UID: \"73900785-3360-4c42-8a5d-8c0e258fc637\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg" Nov 03 18:15:00.462275 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.462251 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60483b95-19d2-46c3-ad70-5d8b36ffb572-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp\" (UID: \"60483b95-19d2-46c3-ad70-5d8b36ffb572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp" Nov 03 18:15:00.462449 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.462432 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78a7ba2-1042-492c-b52c-dca73afd27d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v\" (UID: \"b78a7ba2-1042-492c-b52c-dca73afd27d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v" Nov 03 18:15:00.489101 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.489080 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz" Nov 03 18:15:00.519286 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.519262 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48jrd\" (UniqueName: \"kubernetes.io/projected/73900785-3360-4c42-8a5d-8c0e258fc637-kube-api-access-48jrd\") pod \"maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg\" (UID: \"73900785-3360-4c42-8a5d-8c0e258fc637\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg" Nov 03 18:15:00.519597 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.519578 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqxzm\" (UniqueName: \"kubernetes.io/projected/b78a7ba2-1042-492c-b52c-dca73afd27d5-kube-api-access-tqxzm\") pod \"maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v\" (UID: \"b78a7ba2-1042-492c-b52c-dca73afd27d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v" Nov 03 18:15:00.519706 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.519685 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4crbc\" (UniqueName: \"kubernetes.io/projected/60483b95-19d2-46c3-ad70-5d8b36ffb572-kube-api-access-4crbc\") pod \"maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp\" (UID: \"60483b95-19d2-46c3-ad70-5d8b36ffb572\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp" Nov 03 18:15:00.519749 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.519685 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6v4s\" (UniqueName: \"kubernetes.io/projected/d22875d5-06bf-4abe-9961-9f5c9922eac0-kube-api-access-p6v4s\") pod \"maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf\" (UID: \"d22875d5-06bf-4abe-9961-9f5c9922eac0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf" Nov 03 18:15:00.535357 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.535337 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp" Nov 03 18:15:00.542200 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.542179 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v" Nov 03 18:15:00.684155 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.684112 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:00.808647 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.808586 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg" Nov 03 18:15:00.814801 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.814783 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf" Nov 03 18:15:00.875817 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.875441 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz"] Nov 03 18:15:00.891200 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.890957 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:00.990987 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.990957 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp"] Nov 03 18:15:00.991139 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:00.991019 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v"] Nov 03 18:15:01.004985 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:15:01.004959 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60483b95_19d2_46c3_ad70_5d8b36ffb572.slice/crio-c886b8e57e32861a7dea6d3b46111f083f7736ae5484fb1be5c1180adef0b773 WatchSource:0}: Error finding container c886b8e57e32861a7dea6d3b46111f083f7736ae5484fb1be5c1180adef0b773: Status 404 returned error can't find the container with id c886b8e57e32861a7dea6d3b46111f083f7736ae5484fb1be5c1180adef0b773 Nov 03 18:15:01.123761 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.123717 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:01.284730 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.281728 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg"] Nov 03 18:15:01.287225 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.286656 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf"] Nov 03 18:15:01.289710 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.289686 5805 generic.go:356] "Generic (PLEG): container finished" podID="44286112-623b-4654-9b84-7eed4dd067d9" containerID="a0d08bd0b2d0ab1dc5283ad828636d0c9c8e079345aa0702b8234a2cb6485127" exitCode=0 Nov 03 18:15:01.289807 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.289757 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" event={"ID":"44286112-623b-4654-9b84-7eed4dd067d9","Type":"ContainerDied","Data":"a0d08bd0b2d0ab1dc5283ad828636d0c9c8e079345aa0702b8234a2cb6485127"} Nov 03 18:15:01.294965 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.294494 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v" event={"ID":"b78a7ba2-1042-492c-b52c-dca73afd27d5","Type":"ContainerStarted","Data":"7f06148ea960aeb2d76307c4fe7fdd261ef3227a307dac4ec51f81a8d69ac03f"} Nov 03 18:15:01.315703 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.315673 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz" event={"ID":"7221deb2-4d2f-4f43-b7a5-064464fc3991","Type":"ContainerStarted","Data":"d3051418faae196b518e3886b454dabd0fbd5b8ff13aba6aec8ed3cf00ca5022"} Nov 03 18:15:01.317819 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.317793 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp" event={"ID":"60483b95-19d2-46c3-ad70-5d8b36ffb572","Type":"ContainerStarted","Data":"c886b8e57e32861a7dea6d3b46111f083f7736ae5484fb1be5c1180adef0b773"} Nov 03 18:15:01.321185 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.321159 5805 generic.go:356] "Generic (PLEG): container finished" podID="d5030e39-1c83-41e5-9484-f0b9c183084d" containerID="9f205abcc96a42eec8883c6dbb6cc8b779fa6d702a2750adf4bee81d89fd0c71" exitCode=0 Nov 03 18:15:01.321283 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.321240 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" event={"ID":"d5030e39-1c83-41e5-9484-f0b9c183084d","Type":"ContainerDied","Data":"9f205abcc96a42eec8883c6dbb6cc8b779fa6d702a2750adf4bee81d89fd0c71"} Nov 03 18:15:01.326921 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.326722 5805 generic.go:356] "Generic (PLEG): container finished" podID="348445a7-ec7a-454d-86aa-8ef0beeccb33" containerID="265c19c926adecfebfcbc5c7d777a8aab146a3624b3db170f0915a6532746351" exitCode=0 Nov 03 18:15:01.326921 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.326781 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" event={"ID":"348445a7-ec7a-454d-86aa-8ef0beeccb33","Type":"ContainerDied","Data":"265c19c926adecfebfcbc5c7d777a8aab146a3624b3db170f0915a6532746351"} Nov 03 18:15:01.352294 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.352266 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:15:01.405354 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.405330 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:15:01.416318 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.416299 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:15:01.533665 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.533642 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-istio-envoy\") pod \"44286112-623b-4654-9b84-7eed4dd067d9\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " Nov 03 18:15:01.533774 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.533696 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44286112-623b-4654-9b84-7eed4dd067d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44286112-623b-4654-9b84-7eed4dd067d9-istiod-ca-cert\") pod \"44286112-623b-4654-9b84-7eed4dd067d9\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " Nov 03 18:15:01.533774 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.533720 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5030e39-1c83-41e5-9484-f0b9c183084d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5030e39-1c83-41e5-9484-f0b9c183084d-istiod-ca-cert\") pod \"d5030e39-1c83-41e5-9484-f0b9c183084d\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " Nov 03 18:15:01.533774 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.533738 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-data\") pod \"348445a7-ec7a-454d-86aa-8ef0beeccb33\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " Nov 03 18:15:01.533892 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.533776 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-podinfo\") pod \"d5030e39-1c83-41e5-9484-f0b9c183084d\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " Nov 03 18:15:01.533892 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.533797 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-workload-socket\") pod \"d5030e39-1c83-41e5-9484-f0b9c183084d\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " Nov 03 18:15:01.533892 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.533816 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-credential-socket\") pod \"348445a7-ec7a-454d-86aa-8ef0beeccb33\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " Nov 03 18:15:01.533892 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.533843 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-data\") pod \"d5030e39-1c83-41e5-9484-f0b9c183084d\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " Nov 03 18:15:01.533892 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.533860 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-workload-certs\") pod \"348445a7-ec7a-454d-86aa-8ef0beeccb33\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " Nov 03 18:15:01.533892 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.533879 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-envoy\") pod \"348445a7-ec7a-454d-86aa-8ef0beeccb33\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " Nov 03 18:15:01.534102 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.533909 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44286112-623b-4654-9b84-7eed4dd067d9-istio-token\" (UniqueName: \"kubernetes.io/projected/44286112-623b-4654-9b84-7eed4dd067d9-istio-token\") pod \"44286112-623b-4654-9b84-7eed4dd067d9\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " Nov 03 18:15:01.534102 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.533964 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-token\" (UniqueName: \"kubernetes.io/projected/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-token\") pod \"348445a7-ec7a-454d-86aa-8ef0beeccb33\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " Nov 03 18:15:01.534102 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.533995 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/348445a7-ec7a-454d-86aa-8ef0beeccb33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/348445a7-ec7a-454d-86aa-8ef0beeccb33-istiod-ca-cert\") pod \"348445a7-ec7a-454d-86aa-8ef0beeccb33\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " Nov 03 18:15:01.534102 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.534024 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-istio-data\") pod \"44286112-623b-4654-9b84-7eed4dd067d9\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " Nov 03 18:15:01.534102 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.534047 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-credential-socket\") pod \"44286112-623b-4654-9b84-7eed4dd067d9\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " Nov 03 18:15:01.534318 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.534140 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-workload-socket\") pod \"44286112-623b-4654-9b84-7eed4dd067d9\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " Nov 03 18:15:01.534463 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.534440 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-workload-socket" pod "44286112-623b-4654-9b84-7eed4dd067d9" (UID: "44286112-623b-4654-9b84-7eed4dd067d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:01.534538 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.534440 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-workload-socket" pod "d5030e39-1c83-41e5-9484-f0b9c183084d" (UID: "d5030e39-1c83-41e5-9484-f0b9c183084d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:01.534661 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.534641 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-data" pod "d5030e39-1c83-41e5-9484-f0b9c183084d" (UID: "d5030e39-1c83-41e5-9484-f0b9c183084d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:01.534748 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.534712 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-istio-data" pod "44286112-623b-4654-9b84-7eed4dd067d9" (UID: "44286112-623b-4654-9b84-7eed4dd067d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:01.534849 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.534797 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-data" pod "348445a7-ec7a-454d-86aa-8ef0beeccb33" (UID: "348445a7-ec7a-454d-86aa-8ef0beeccb33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:01.534962 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.534941 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-credential-socket" pod "44286112-623b-4654-9b84-7eed4dd067d9" (UID: "44286112-623b-4654-9b84-7eed4dd067d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:01.535791 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.535061 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-credential-socket" pod "348445a7-ec7a-454d-86aa-8ef0beeccb33" (UID: "348445a7-ec7a-454d-86aa-8ef0beeccb33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:01.535791 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.534848 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/348445a7-ec7a-454d-86aa-8ef0beeccb33-istiod-ca-cert" pod "348445a7-ec7a-454d-86aa-8ef0beeccb33" (UID: "348445a7-ec7a-454d-86aa-8ef0beeccb33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:01.535791 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.535144 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-workload-certs" pod "348445a7-ec7a-454d-86aa-8ef0beeccb33" (UID: "348445a7-ec7a-454d-86aa-8ef0beeccb33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:01.535791 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.535422 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5030e39-1c83-41e5-9484-f0b9c183084d-istiod-ca-cert" pod "d5030e39-1c83-41e5-9484-f0b9c183084d" (UID: "d5030e39-1c83-41e5-9484-f0b9c183084d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:01.535791 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.535467 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/348445a7-ec7a-454d-86aa-8ef0beeccb33-kube-api-access-pmp9c\" (UniqueName: \"kubernetes.io/projected/348445a7-ec7a-454d-86aa-8ef0beeccb33-kube-api-access-pmp9c\") pod \"348445a7-ec7a-454d-86aa-8ef0beeccb33\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " Nov 03 18:15:01.535791 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.535502 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44286112-623b-4654-9b84-7eed4dd067d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44286112-623b-4654-9b84-7eed4dd067d9-istio-podinfo\") pod \"44286112-623b-4654-9b84-7eed4dd067d9\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " Nov 03 18:15:01.535791 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.535538 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44286112-623b-4654-9b84-7eed4dd067d9-istiod-ca-cert" pod "44286112-623b-4654-9b84-7eed4dd067d9" (UID: "44286112-623b-4654-9b84-7eed4dd067d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:01.535791 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.535561 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-podinfo\") pod \"348445a7-ec7a-454d-86aa-8ef0beeccb33\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " Nov 03 18:15:01.535791 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.535589 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-workload-socket\") pod \"348445a7-ec7a-454d-86aa-8ef0beeccb33\" (UID: \"348445a7-ec7a-454d-86aa-8ef0beeccb33\") " Nov 03 18:15:01.535791 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.535617 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-token\" (UniqueName: \"kubernetes.io/projected/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-token\") pod \"d5030e39-1c83-41e5-9484-f0b9c183084d\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " Nov 03 18:15:01.535791 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.535659 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5030e39-1c83-41e5-9484-f0b9c183084d-kube-api-access-l66j5\" (UniqueName: \"kubernetes.io/projected/d5030e39-1c83-41e5-9484-f0b9c183084d-kube-api-access-l66j5\") pod \"d5030e39-1c83-41e5-9484-f0b9c183084d\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " Nov 03 18:15:01.535791 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.535689 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-credential-socket\") pod \"d5030e39-1c83-41e5-9484-f0b9c183084d\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " Nov 03 18:15:01.535791 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.535727 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-workload-certs\") pod \"d5030e39-1c83-41e5-9484-f0b9c183084d\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " Nov 03 18:15:01.535791 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.535775 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-envoy\") pod \"d5030e39-1c83-41e5-9484-f0b9c183084d\" (UID: \"d5030e39-1c83-41e5-9484-f0b9c183084d\") " Nov 03 18:15:01.536802 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.535810 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44286112-623b-4654-9b84-7eed4dd067d9-kube-api-access-ndc28\" (UniqueName: \"kubernetes.io/projected/44286112-623b-4654-9b84-7eed4dd067d9-kube-api-access-ndc28\") pod \"44286112-623b-4654-9b84-7eed4dd067d9\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " Nov 03 18:15:01.536802 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.535839 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-workload-certs\") pod \"44286112-623b-4654-9b84-7eed4dd067d9\" (UID: \"44286112-623b-4654-9b84-7eed4dd067d9\") " Nov 03 18:15:01.536802 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.536300 5805 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-workload-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.536802 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.536314 5805 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44286112-623b-4654-9b84-7eed4dd067d9-istiod-ca-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.536802 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.536323 5805 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5030e39-1c83-41e5-9484-f0b9c183084d-istiod-ca-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.536802 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.536332 5805 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-data\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.536802 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.536341 5805 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-workload-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.536802 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.536350 5805 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-credential-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.536802 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.536358 5805 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-data\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.536802 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.536366 5805 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-workload-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.536802 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.536376 5805 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/348445a7-ec7a-454d-86aa-8ef0beeccb33-istiod-ca-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.536802 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.536384 5805 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-istio-data\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.536802 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.536408 5805 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-credential-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.536802 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.536639 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-workload-certs" pod "44286112-623b-4654-9b84-7eed4dd067d9" (UID: "44286112-623b-4654-9b84-7eed4dd067d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:01.537850 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.537829 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-workload-socket" pod "348445a7-ec7a-454d-86aa-8ef0beeccb33" (UID: "348445a7-ec7a-454d-86aa-8ef0beeccb33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:01.538021 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.537999 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-credential-socket" pod "d5030e39-1c83-41e5-9484-f0b9c183084d" (UID: "d5030e39-1c83-41e5-9484-f0b9c183084d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:01.538173 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.538061 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-workload-certs" pod "d5030e39-1c83-41e5-9484-f0b9c183084d" (UID: "d5030e39-1c83-41e5-9484-f0b9c183084d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:01.569947 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.569900 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:01.628508 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.628383 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-istio-envoy" pod "44286112-623b-4654-9b84-7eed4dd067d9" (UID: "44286112-623b-4654-9b84-7eed4dd067d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:01.629165 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.629002 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-envoy" pod "348445a7-ec7a-454d-86aa-8ef0beeccb33" (UID: "348445a7-ec7a-454d-86aa-8ef0beeccb33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:01.629165 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.629104 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44286112-623b-4654-9b84-7eed4dd067d9-istio-token" pod "44286112-623b-4654-9b84-7eed4dd067d9" (UID: "44286112-623b-4654-9b84-7eed4dd067d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:01.629353 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.629175 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348445a7-ec7a-454d-86aa-8ef0beeccb33-kube-api-access-pmp9c" pod "348445a7-ec7a-454d-86aa-8ef0beeccb33" (UID: "348445a7-ec7a-454d-86aa-8ef0beeccb33"). InnerVolumeSpecName "kube-api-access-pmp9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:01.629353 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.629189 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-podinfo" pod "348445a7-ec7a-454d-86aa-8ef0beeccb33" (UID: "348445a7-ec7a-454d-86aa-8ef0beeccb33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:15:01.629353 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.629267 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44286112-623b-4654-9b84-7eed4dd067d9-istio-podinfo" pod "44286112-623b-4654-9b84-7eed4dd067d9" (UID: "44286112-623b-4654-9b84-7eed4dd067d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:15:01.629668 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.629645 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-envoy" pod "d5030e39-1c83-41e5-9484-f0b9c183084d" (UID: "d5030e39-1c83-41e5-9484-f0b9c183084d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:01.629752 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.629675 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-token" pod "d5030e39-1c83-41e5-9484-f0b9c183084d" (UID: "d5030e39-1c83-41e5-9484-f0b9c183084d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:01.629752 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.629703 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-podinfo" pod "d5030e39-1c83-41e5-9484-f0b9c183084d" (UID: "d5030e39-1c83-41e5-9484-f0b9c183084d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:15:01.629982 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.629966 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5030e39-1c83-41e5-9484-f0b9c183084d-kube-api-access-l66j5" pod "d5030e39-1c83-41e5-9484-f0b9c183084d" (UID: "d5030e39-1c83-41e5-9484-f0b9c183084d"). InnerVolumeSpecName "kube-api-access-l66j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:01.630221 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.630207 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-token" pod "348445a7-ec7a-454d-86aa-8ef0beeccb33" (UID: "348445a7-ec7a-454d-86aa-8ef0beeccb33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:01.630572 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.630553 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44286112-623b-4654-9b84-7eed4dd067d9-kube-api-access-ndc28" pod "44286112-623b-4654-9b84-7eed4dd067d9" (UID: "44286112-623b-4654-9b84-7eed4dd067d9"). InnerVolumeSpecName "kube-api-access-ndc28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:01.637180 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.637156 5805 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-podinfo\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.637279 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.637183 5805 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-workload-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.637279 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.637198 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.637279 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.637212 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l66j5\" (UniqueName: \"kubernetes.io/projected/d5030e39-1c83-41e5-9484-f0b9c183084d-kube-api-access-l66j5\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.637279 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.637227 5805 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-credential-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.637279 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.637240 5805 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-workload-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.637279 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.637253 5805 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-envoy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.637279 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.637266 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndc28\" (UniqueName: \"kubernetes.io/projected/44286112-623b-4654-9b84-7eed4dd067d9-kube-api-access-ndc28\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.637577 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.637280 5805 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-workload-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.637577 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.637293 5805 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44286112-623b-4654-9b84-7eed4dd067d9-istio-envoy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.637577 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.637307 5805 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5030e39-1c83-41e5-9484-f0b9c183084d-istio-podinfo\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.637577 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.637320 5805 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-envoy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.637577 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.637334 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44286112-623b-4654-9b84-7eed4dd067d9-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.637577 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.637347 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/348445a7-ec7a-454d-86aa-8ef0beeccb33-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.637577 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.637360 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmp9c\" (UniqueName: \"kubernetes.io/projected/348445a7-ec7a-454d-86aa-8ef0beeccb33-kube-api-access-pmp9c\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:01.637577 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:01.637374 5805 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44286112-623b-4654-9b84-7eed4dd067d9-istio-podinfo\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:02.336005 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.335977 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" Nov 03 18:15:02.336381 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.335976 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4" event={"ID":"44286112-623b-4654-9b84-7eed4dd067d9","Type":"ContainerDied","Data":"a3b18099a3eaf82b995f2e533a30c0dabddda68b79f021beab499bc3af514263"} Nov 03 18:15:02.336381 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.336131 5805 scope.go:122] "RemoveContainer" containerID="a0d08bd0b2d0ab1dc5283ad828636d0c9c8e079345aa0702b8234a2cb6485127" Nov 03 18:15:02.338021 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.337995 5805 generic.go:356] "Generic (PLEG): container finished" podID="b78a7ba2-1042-492c-b52c-dca73afd27d5" containerID="7ac7733ec467bd1e7ab1078fa6060f657520b9250cd140ad5bcbf0e7b22b0f6c" exitCode=60 Nov 03 18:15:02.338133 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.338079 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v" event={"ID":"b78a7ba2-1042-492c-b52c-dca73afd27d5","Type":"ContainerDied","Data":"7ac7733ec467bd1e7ab1078fa6060f657520b9250cd140ad5bcbf0e7b22b0f6c"} Nov 03 18:15:02.338481 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.338383 5805 scope.go:122] "RemoveContainer" containerID="7ac7733ec467bd1e7ab1078fa6060f657520b9250cd140ad5bcbf0e7b22b0f6c" Nov 03 18:15:02.341543 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.341508 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf" event={"ID":"d22875d5-06bf-4abe-9961-9f5c9922eac0","Type":"ContainerStarted","Data":"3571c6b9af22ee20506f9551cd3692cf65b74035f3fbf673f58a3207ac500c61"} Nov 03 18:15:02.341632 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.341560 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf" event={"ID":"d22875d5-06bf-4abe-9961-9f5c9922eac0","Type":"ContainerStarted","Data":"8d5dc1687bd4cdaebe042ed047df02e404e110574c4c9a8eb7ac710c677973af"} Nov 03 18:15:02.343222 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.343200 5805 generic.go:356] "Generic (PLEG): container finished" podID="7221deb2-4d2f-4f43-b7a5-064464fc3991" containerID="5e998c6eb145eb84a64a4cc18f4c5e6346eb6b8ad14acfaf6f3fc53e2e0c3ff8" exitCode=60 Nov 03 18:15:02.343491 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.343469 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz" event={"ID":"7221deb2-4d2f-4f43-b7a5-064464fc3991","Type":"ContainerDied","Data":"5e998c6eb145eb84a64a4cc18f4c5e6346eb6b8ad14acfaf6f3fc53e2e0c3ff8"} Nov 03 18:15:02.343873 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.343583 5805 scope.go:122] "RemoveContainer" containerID="5e998c6eb145eb84a64a4cc18f4c5e6346eb6b8ad14acfaf6f3fc53e2e0c3ff8" Nov 03 18:15:02.345878 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.345567 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg" event={"ID":"73900785-3360-4c42-8a5d-8c0e258fc637","Type":"ContainerStarted","Data":"08892b0c8a555bda9bb03abbb7f6ac3520eba8501a5afe9967b04643efa6539d"} Nov 03 18:15:02.345878 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.345612 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg" event={"ID":"73900785-3360-4c42-8a5d-8c0e258fc637","Type":"ContainerStarted","Data":"58dae839baec8a297b9afdc93e21e788462ec3d3dc772ce94055b9ec3a14de12"} Nov 03 18:15:02.345878 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.345753 5805 scope.go:122] "RemoveContainer" containerID="08892b0c8a555bda9bb03abbb7f6ac3520eba8501a5afe9967b04643efa6539d" Nov 03 18:15:02.347436 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.347415 5805 generic.go:356] "Generic (PLEG): container finished" podID="60483b95-19d2-46c3-ad70-5d8b36ffb572" containerID="4b1b60f0d1f14bd33e02b977aa6d7c0c40a2bc84362b51c5acbea3cb684bc587" exitCode=60 Nov 03 18:15:02.347553 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.347536 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp" event={"ID":"60483b95-19d2-46c3-ad70-5d8b36ffb572","Type":"ContainerDied","Data":"4b1b60f0d1f14bd33e02b977aa6d7c0c40a2bc84362b51c5acbea3cb684bc587"} Nov 03 18:15:02.347694 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.347681 5805 scope.go:122] "RemoveContainer" containerID="4b1b60f0d1f14bd33e02b977aa6d7c0c40a2bc84362b51c5acbea3cb684bc587" Nov 03 18:15:02.348954 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.348935 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" Nov 03 18:15:02.349050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.348935 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw" event={"ID":"d5030e39-1c83-41e5-9484-f0b9c183084d","Type":"ContainerDied","Data":"0035a44802ae7a4173ee6f963ce9de09b3d350c2259ddf5b804bddf5824f9b8e"} Nov 03 18:15:02.350797 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.350778 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" event={"ID":"348445a7-ec7a-454d-86aa-8ef0beeccb33","Type":"ContainerDied","Data":"3c1b5debeb9e61891c11269908c1edad4c0a02b6d43a549be506f142fe7aa7e4"} Nov 03 18:15:02.350973 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.350857 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm" Nov 03 18:15:02.378594 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.378517 5805 scope.go:122] "RemoveContainer" containerID="9f205abcc96a42eec8883c6dbb6cc8b779fa6d702a2750adf4bee81d89fd0c71" Nov 03 18:15:02.396270 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.396247 5805 scope.go:122] "RemoveContainer" containerID="265c19c926adecfebfcbc5c7d777a8aab146a3624b3db170f0915a6532746351" Nov 03 18:15:02.495467 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.495423 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf" podStartSLOduration=2.49540985 podStartE2EDuration="2.49540985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:02.493489716 +0000 UTC m=+1739.627118727" watchObservedRunningTime="2026-11-03 18:15:02.49540985 +0000 UTC m=+1739.629038854" Nov 03 18:15:02.706847 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.706813 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw"] Nov 03 18:15:02.713608 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.713580 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-fjwpw"] Nov 03 18:15:02.816197 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.816161 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm"] Nov 03 18:15:02.820230 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.820199 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-956zm"] Nov 03 18:15:02.978614 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.978545 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="348445a7-ec7a-454d-86aa-8ef0beeccb33" path="/var/lib/kubelet/pods/348445a7-ec7a-454d-86aa-8ef0beeccb33/volumes" Nov 03 18:15:02.979019 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:02.979005 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5030e39-1c83-41e5-9484-f0b9c183084d" path="/var/lib/kubelet/pods/d5030e39-1c83-41e5-9484-f0b9c183084d/volumes" Nov 03 18:15:03.131983 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:03.131929 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:03.145217 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:03.145189 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4"] Nov 03 18:15:03.150539 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:03.150508 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-0ccbb9-openshift-default-6f444f99bc-rvbj4"] Nov 03 18:15:03.359502 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:03.359465 5805 generic.go:356] "Generic (PLEG): container finished" podID="b78a7ba2-1042-492c-b52c-dca73afd27d5" containerID="ed2d43f49db1aba0238a83b079fc844e18838c4615e38327de538e18cde47f64" exitCode=60 Nov 03 18:15:03.359879 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:03.359507 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v" event={"ID":"b78a7ba2-1042-492c-b52c-dca73afd27d5","Type":"ContainerDied","Data":"ed2d43f49db1aba0238a83b079fc844e18838c4615e38327de538e18cde47f64"} Nov 03 18:15:03.359879 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:03.359578 5805 scope.go:122] "RemoveContainer" containerID="7ac7733ec467bd1e7ab1078fa6060f657520b9250cd140ad5bcbf0e7b22b0f6c" Nov 03 18:15:03.359879 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:03.359847 5805 scope.go:122] "RemoveContainer" containerID="ed2d43f49db1aba0238a83b079fc844e18838c4615e38327de538e18cde47f64" Nov 03 18:15:03.360259 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:15:03.360212 5805 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v_odh-ai-gateway-infra(b78a7ba2-1042-492c-b52c-dca73afd27d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v" podUID="b78a7ba2-1042-492c-b52c-dca73afd27d5" Nov 03 18:15:03.361636 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:03.361614 5805 generic.go:356] "Generic (PLEG): container finished" podID="d22875d5-06bf-4abe-9961-9f5c9922eac0" containerID="3571c6b9af22ee20506f9551cd3692cf65b74035f3fbf673f58a3207ac500c61" exitCode=60 Nov 03 18:15:03.361734 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:03.361703 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf" event={"ID":"d22875d5-06bf-4abe-9961-9f5c9922eac0","Type":"ContainerDied","Data":"3571c6b9af22ee20506f9551cd3692cf65b74035f3fbf673f58a3207ac500c61"} Nov 03 18:15:03.361930 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:03.361913 5805 scope.go:122] "RemoveContainer" containerID="3571c6b9af22ee20506f9551cd3692cf65b74035f3fbf673f58a3207ac500c61" Nov 03 18:15:03.372617 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:03.372491 5805 generic.go:356] "Generic (PLEG): container finished" podID="7221deb2-4d2f-4f43-b7a5-064464fc3991" containerID="f9e53ef01d8bfd70d46d659eac381d98570c8f4e6dda241c0d68ff6788f425b4" exitCode=60 Nov 03 18:15:03.372617 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:03.372584 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz" event={"ID":"7221deb2-4d2f-4f43-b7a5-064464fc3991","Type":"ContainerDied","Data":"f9e53ef01d8bfd70d46d659eac381d98570c8f4e6dda241c0d68ff6788f425b4"} Nov 03 18:15:03.372957 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:03.372780 5805 scope.go:122] "RemoveContainer" containerID="f9e53ef01d8bfd70d46d659eac381d98570c8f4e6dda241c0d68ff6788f425b4" Nov 03 18:15:03.373026 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:15:03.372978 5805 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz_odh-ai-gateway-infra(7221deb2-4d2f-4f43-b7a5-064464fc3991)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz" podUID="7221deb2-4d2f-4f43-b7a5-064464fc3991" Nov 03 18:15:03.374506 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:03.374484 5805 generic.go:356] "Generic (PLEG): container finished" podID="73900785-3360-4c42-8a5d-8c0e258fc637" containerID="08892b0c8a555bda9bb03abbb7f6ac3520eba8501a5afe9967b04643efa6539d" exitCode=60 Nov 03 18:15:03.374604 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:03.374572 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg" event={"ID":"73900785-3360-4c42-8a5d-8c0e258fc637","Type":"ContainerDied","Data":"08892b0c8a555bda9bb03abbb7f6ac3520eba8501a5afe9967b04643efa6539d"} Nov 03 18:15:03.374670 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:03.374616 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg" event={"ID":"73900785-3360-4c42-8a5d-8c0e258fc637","Type":"ContainerStarted","Data":"737c519f22e8379fdbc889b6e34a6a3bdf45adaa84da840498282df894675f3b"} Nov 03 18:15:03.376257 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:03.376237 5805 generic.go:356] "Generic (PLEG): container finished" podID="60483b95-19d2-46c3-ad70-5d8b36ffb572" containerID="7c960c0383e76292a35de7a1b45ff9c5b56799c7407d40214a9eb2c03a291136" exitCode=60 Nov 03 18:15:03.376355 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:03.376316 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp" event={"ID":"60483b95-19d2-46c3-ad70-5d8b36ffb572","Type":"ContainerDied","Data":"7c960c0383e76292a35de7a1b45ff9c5b56799c7407d40214a9eb2c03a291136"} Nov 03 18:15:03.376528 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:03.376515 5805 scope.go:122] "RemoveContainer" containerID="7c960c0383e76292a35de7a1b45ff9c5b56799c7407d40214a9eb2c03a291136" Nov 03 18:15:03.376716 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:15:03.376696 5805 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp_odh-ai-gateway-infra(60483b95-19d2-46c3-ad70-5d8b36ffb572)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp" podUID="60483b95-19d2-46c3-ad70-5d8b36ffb572" Nov 03 18:15:03.420115 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:03.420088 5805 scope.go:122] "RemoveContainer" containerID="5e998c6eb145eb84a64a4cc18f4c5e6346eb6b8ad14acfaf6f3fc53e2e0c3ff8" Nov 03 18:15:03.503008 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:03.502988 5805 scope.go:122] "RemoveContainer" containerID="4b1b60f0d1f14bd33e02b977aa6d7c0c40a2bc84362b51c5acbea3cb684bc587" Nov 03 18:15:03.795150 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:03.795067 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg" podStartSLOduration=3.795051925 podStartE2EDuration="3.795051925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:03.792661641 +0000 UTC m=+1740.926290660" watchObservedRunningTime="2026-11-03 18:15:03.795051925 +0000 UTC m=+1740.928680959" Nov 03 18:15:04.387576 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:04.387549 5805 scope.go:122] "RemoveContainer" containerID="f9e53ef01d8bfd70d46d659eac381d98570c8f4e6dda241c0d68ff6788f425b4" Nov 03 18:15:04.387989 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:15:04.387895 5805 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz_odh-ai-gateway-infra(7221deb2-4d2f-4f43-b7a5-064464fc3991)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz" podUID="7221deb2-4d2f-4f43-b7a5-064464fc3991" Nov 03 18:15:04.389839 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:04.389810 5805 generic.go:356] "Generic (PLEG): container finished" podID="73900785-3360-4c42-8a5d-8c0e258fc637" containerID="737c519f22e8379fdbc889b6e34a6a3bdf45adaa84da840498282df894675f3b" exitCode=60 Nov 03 18:15:04.389939 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:04.389887 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg" event={"ID":"73900785-3360-4c42-8a5d-8c0e258fc637","Type":"ContainerDied","Data":"737c519f22e8379fdbc889b6e34a6a3bdf45adaa84da840498282df894675f3b"} Nov 03 18:15:04.390002 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:04.389940 5805 scope.go:122] "RemoveContainer" containerID="08892b0c8a555bda9bb03abbb7f6ac3520eba8501a5afe9967b04643efa6539d" Nov 03 18:15:04.390114 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:04.390101 5805 scope.go:122] "RemoveContainer" containerID="737c519f22e8379fdbc889b6e34a6a3bdf45adaa84da840498282df894675f3b" Nov 03 18:15:04.391301 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:15:04.391268 5805 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg_odh-ai-gateway-infra(73900785-3360-4c42-8a5d-8c0e258fc637)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg" podUID="73900785-3360-4c42-8a5d-8c0e258fc637" Nov 03 18:15:04.401632 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:04.401613 5805 scope.go:122] "RemoveContainer" containerID="7c960c0383e76292a35de7a1b45ff9c5b56799c7407d40214a9eb2c03a291136" Nov 03 18:15:04.401910 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:15:04.401872 5805 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp_odh-ai-gateway-infra(60483b95-19d2-46c3-ad70-5d8b36ffb572)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp" podUID="60483b95-19d2-46c3-ad70-5d8b36ffb572" Nov 03 18:15:04.403566 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:04.403545 5805 scope.go:122] "RemoveContainer" containerID="ed2d43f49db1aba0238a83b079fc844e18838c4615e38327de538e18cde47f64" Nov 03 18:15:04.403805 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:15:04.403784 5805 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v_odh-ai-gateway-infra(b78a7ba2-1042-492c-b52c-dca73afd27d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v" podUID="b78a7ba2-1042-492c-b52c-dca73afd27d5" Nov 03 18:15:04.405178 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:04.405152 5805 generic.go:356] "Generic (PLEG): container finished" podID="d22875d5-06bf-4abe-9961-9f5c9922eac0" containerID="6b663cc90cd3187c49fe1a0bc949033495802267cc0af99e8f52c04e4235f075" exitCode=60 Nov 03 18:15:04.405279 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:04.405180 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf" event={"ID":"d22875d5-06bf-4abe-9961-9f5c9922eac0","Type":"ContainerDied","Data":"6b663cc90cd3187c49fe1a0bc949033495802267cc0af99e8f52c04e4235f075"} Nov 03 18:15:04.405455 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:04.405439 5805 scope.go:122] "RemoveContainer" containerID="6b663cc90cd3187c49fe1a0bc949033495802267cc0af99e8f52c04e4235f075" Nov 03 18:15:04.405654 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:15:04.405636 5805 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf_odh-ai-gateway-infra(d22875d5-06bf-4abe-9961-9f5c9922eac0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf" podUID="d22875d5-06bf-4abe-9961-9f5c9922eac0" Nov 03 18:15:04.414452 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:04.414427 5805 scope.go:122] "RemoveContainer" containerID="3571c6b9af22ee20506f9551cd3692cf65b74035f3fbf673f58a3207ac500c61" Nov 03 18:15:04.978018 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:04.977980 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44286112-623b-4654-9b84-7eed4dd067d9" path="/var/lib/kubelet/pods/44286112-623b-4654-9b84-7eed4dd067d9/volumes" Nov 03 18:15:05.413782 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:05.413750 5805 scope.go:122] "RemoveContainer" containerID="6b663cc90cd3187c49fe1a0bc949033495802267cc0af99e8f52c04e4235f075" Nov 03 18:15:05.414226 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:15:05.414095 5805 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf_odh-ai-gateway-infra(d22875d5-06bf-4abe-9961-9f5c9922eac0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf" podUID="d22875d5-06bf-4abe-9961-9f5c9922eac0" Nov 03 18:15:05.415496 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:05.415475 5805 scope.go:122] "RemoveContainer" containerID="737c519f22e8379fdbc889b6e34a6a3bdf45adaa84da840498282df894675f3b" Nov 03 18:15:05.415716 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:15:05.415698 5805 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg_odh-ai-gateway-infra(73900785-3360-4c42-8a5d-8c0e258fc637)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg" podUID="73900785-3360-4c42-8a5d-8c0e258fc637" Nov 03 18:15:06.466767 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:06.466732 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn"] Nov 03 18:15:06.515162 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:06.515131 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl"] Nov 03 18:15:06.631295 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:06.631260 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:07.858990 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:07.858954 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:09.660762 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:09.660730 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:12.224242 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.224211 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9525b615-wtwgt"] Nov 03 18:15:12.224840 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.224825 5805 cpu_manager.go:414] "containerMap: removing container" podUID="d5030e39-1c83-41e5-9484-f0b9c183084d" containerName="istio-proxy" Nov 03 18:15:12.224920 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.224841 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5030e39-1c83-41e5-9484-f0b9c183084d" containerName="istio-proxy" Nov 03 18:15:12.224920 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.224866 5805 cpu_manager.go:414] "containerMap: removing container" podUID="44286112-623b-4654-9b84-7eed4dd067d9" containerName="istio-proxy" Nov 03 18:15:12.224920 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.224872 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44286112-623b-4654-9b84-7eed4dd067d9" containerName="istio-proxy" Nov 03 18:15:12.224920 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.224899 5805 cpu_manager.go:414] "containerMap: removing container" podUID="348445a7-ec7a-454d-86aa-8ef0beeccb33" containerName="istio-proxy" Nov 03 18:15:12.224920 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.224906 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="348445a7-ec7a-454d-86aa-8ef0beeccb33" containerName="istio-proxy" Nov 03 18:15:12.225073 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.225006 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="348445a7-ec7a-454d-86aa-8ef0beeccb33" containerName="istio-proxy" Nov 03 18:15:12.225073 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.225029 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5030e39-1c83-41e5-9484-f0b9c183084d" containerName="istio-proxy" Nov 03 18:15:12.225073 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.225046 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="44286112-623b-4654-9b84-7eed4dd067d9" containerName="istio-proxy" Nov 03 18:15:12.229672 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.229651 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9525b615-wtwgt" Nov 03 18:15:12.283846 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.283812 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9525b615-wtwgt"] Nov 03 18:15:12.328577 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.328548 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:15:12.387341 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.387302 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht8x6\" (UniqueName: \"kubernetes.io/projected/00a224da-b74e-4186-b7ef-030cc5fc1c92-kube-api-access-ht8x6\") pod \"maas-api-revoke-sub-9525b615-wtwgt\" (UID: \"00a224da-b74e-4186-b7ef-030cc5fc1c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9525b615-wtwgt" Nov 03 18:15:12.387341 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.387347 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a224da-b74e-4186-b7ef-030cc5fc1c92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9525b615-wtwgt\" (UID: \"00a224da-b74e-4186-b7ef-030cc5fc1c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9525b615-wtwgt" Nov 03 18:15:12.387602 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.387526 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a224da-b74e-4186-b7ef-030cc5fc1c92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9525b615-wtwgt\" (UID: \"00a224da-b74e-4186-b7ef-030cc5fc1c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9525b615-wtwgt" Nov 03 18:15:12.472796 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.472766 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:12.489588 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.489502 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a224da-b74e-4186-b7ef-030cc5fc1c92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9525b615-wtwgt\" (UID: \"00a224da-b74e-4186-b7ef-030cc5fc1c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9525b615-wtwgt" Nov 03 18:15:12.489734 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.489717 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht8x6\" (UniqueName: \"kubernetes.io/projected/00a224da-b74e-4186-b7ef-030cc5fc1c92-kube-api-access-ht8x6\") pod \"maas-api-revoke-sub-9525b615-wtwgt\" (UID: \"00a224da-b74e-4186-b7ef-030cc5fc1c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9525b615-wtwgt" Nov 03 18:15:12.489799 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.489769 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a224da-b74e-4186-b7ef-030cc5fc1c92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9525b615-wtwgt\" (UID: \"00a224da-b74e-4186-b7ef-030cc5fc1c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9525b615-wtwgt" Nov 03 18:15:12.490583 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.490553 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a224da-b74e-4186-b7ef-030cc5fc1c92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9525b615-wtwgt\" (UID: \"00a224da-b74e-4186-b7ef-030cc5fc1c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9525b615-wtwgt" Nov 03 18:15:12.492456 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.492430 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a224da-b74e-4186-b7ef-030cc5fc1c92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9525b615-wtwgt\" (UID: \"00a224da-b74e-4186-b7ef-030cc5fc1c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9525b615-wtwgt" Nov 03 18:15:12.547673 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.538859 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp"] Nov 03 18:15:12.627870 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.627807 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht8x6\" (UniqueName: \"kubernetes.io/projected/00a224da-b74e-4186-b7ef-030cc5fc1c92-kube-api-access-ht8x6\") pod \"maas-api-revoke-sub-9525b615-wtwgt\" (UID: \"00a224da-b74e-4186-b7ef-030cc5fc1c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9525b615-wtwgt" Nov 03 18:15:12.741776 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.741704 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp" Nov 03 18:15:12.846986 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.846952 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9525b615-wtwgt" Nov 03 18:15:12.895609 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.895559 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60483b95-19d2-46c3-ad70-5d8b36ffb572-maas-api-service-ca" pod "60483b95-19d2-46c3-ad70-5d8b36ffb572" (UID: "60483b95-19d2-46c3-ad70-5d8b36ffb572"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:12.902676 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.895654 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60483b95-19d2-46c3-ad70-5d8b36ffb572-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60483b95-19d2-46c3-ad70-5d8b36ffb572-maas-api-service-ca\") pod \"60483b95-19d2-46c3-ad70-5d8b36ffb572\" (UID: \"60483b95-19d2-46c3-ad70-5d8b36ffb572\") " Nov 03 18:15:12.902676 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.895985 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60483b95-19d2-46c3-ad70-5d8b36ffb572-kube-api-access-4crbc\" (UniqueName: \"kubernetes.io/projected/60483b95-19d2-46c3-ad70-5d8b36ffb572-kube-api-access-4crbc\") pod \"60483b95-19d2-46c3-ad70-5d8b36ffb572\" (UID: \"60483b95-19d2-46c3-ad70-5d8b36ffb572\") " Nov 03 18:15:12.902676 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.902457 5805 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60483b95-19d2-46c3-ad70-5d8b36ffb572-maas-api-service-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:12.905460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:12.904703 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60483b95-19d2-46c3-ad70-5d8b36ffb572-kube-api-access-4crbc" pod "60483b95-19d2-46c3-ad70-5d8b36ffb572" (UID: "60483b95-19d2-46c3-ad70-5d8b36ffb572"). InnerVolumeSpecName "kube-api-access-4crbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:13.009238 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:13.009147 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4crbc\" (UniqueName: \"kubernetes.io/projected/60483b95-19d2-46c3-ad70-5d8b36ffb572-kube-api-access-4crbc\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:13.252454 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:13.252421 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9525b615-wtwgt"] Nov 03 18:15:13.258062 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:15:13.258034 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a224da_b74e_4186_b7ef_030cc5fc1c92.slice/crio-cbf7a02b228255c08fa467106967b704f51eb2ea0f2317b604314ddd4c555231 WatchSource:0}: Error finding container cbf7a02b228255c08fa467106967b704f51eb2ea0f2317b604314ddd4c555231: Status 404 returned error can't find the container with id cbf7a02b228255c08fa467106967b704f51eb2ea0f2317b604314ddd4c555231 Nov 03 18:15:13.480418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:13.479462 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp" Nov 03 18:15:13.480418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:13.479465 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp" event={"ID":"60483b95-19d2-46c3-ad70-5d8b36ffb572","Type":"ContainerDied","Data":"c886b8e57e32861a7dea6d3b46111f083f7736ae5484fb1be5c1180adef0b773"} Nov 03 18:15:13.480418 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:13.479549 5805 scope.go:122] "RemoveContainer" containerID="7c960c0383e76292a35de7a1b45ff9c5b56799c7407d40214a9eb2c03a291136" Nov 03 18:15:13.482800 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:13.482723 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9525b615-wtwgt" event={"ID":"00a224da-b74e-4186-b7ef-030cc5fc1c92","Type":"ContainerStarted","Data":"cbf7a02b228255c08fa467106967b704f51eb2ea0f2317b604314ddd4c555231"} Nov 03 18:15:13.633547 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:13.633524 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp"] Nov 03 18:15:13.641167 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:13.641138 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0adcbc-29895495-xrsmp"] Nov 03 18:15:13.738687 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:13.738655 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-712e5ae1-s7d65"] Nov 03 18:15:13.739659 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:13.739631 5805 cpu_manager.go:414] "containerMap: removing container" podUID="60483b95-19d2-46c3-ad70-5d8b36ffb572" containerName="cleanup" Nov 03 18:15:13.739659 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:13.739654 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60483b95-19d2-46c3-ad70-5d8b36ffb572" containerName="cleanup" Nov 03 18:15:13.739912 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:13.739893 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="60483b95-19d2-46c3-ad70-5d8b36ffb572" containerName="cleanup" Nov 03 18:15:13.739969 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:13.739919 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="60483b95-19d2-46c3-ad70-5d8b36ffb572" containerName="cleanup" Nov 03 18:15:13.743511 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:13.743492 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-712e5ae1-s7d65" Nov 03 18:15:13.796530 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:13.796447 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-712e5ae1-s7d65"] Nov 03 18:15:13.845977 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:13.845942 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5323ef-e7f2-4921-bed5-9e71e2aa835d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-712e5ae1-s7d65\" (UID: \"ce5323ef-e7f2-4921-bed5-9e71e2aa835d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712e5ae1-s7d65" Nov 03 18:15:13.846117 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:13.846029 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n6dg\" (UniqueName: \"kubernetes.io/projected/ce5323ef-e7f2-4921-bed5-9e71e2aa835d-kube-api-access-7n6dg\") pod \"maas-api-revoke-sub-712e5ae1-s7d65\" (UID: \"ce5323ef-e7f2-4921-bed5-9e71e2aa835d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712e5ae1-s7d65" Nov 03 18:15:13.846117 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:13.846074 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5323ef-e7f2-4921-bed5-9e71e2aa835d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-712e5ae1-s7d65\" (UID: \"ce5323ef-e7f2-4921-bed5-9e71e2aa835d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712e5ae1-s7d65" Nov 03 18:15:13.947598 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:13.947562 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5323ef-e7f2-4921-bed5-9e71e2aa835d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-712e5ae1-s7d65\" (UID: \"ce5323ef-e7f2-4921-bed5-9e71e2aa835d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712e5ae1-s7d65" Nov 03 18:15:13.947860 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:13.947842 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5323ef-e7f2-4921-bed5-9e71e2aa835d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-712e5ae1-s7d65\" (UID: \"ce5323ef-e7f2-4921-bed5-9e71e2aa835d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712e5ae1-s7d65" Nov 03 18:15:13.947943 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:13.947878 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n6dg\" (UniqueName: \"kubernetes.io/projected/ce5323ef-e7f2-4921-bed5-9e71e2aa835d-kube-api-access-7n6dg\") pod \"maas-api-revoke-sub-712e5ae1-s7d65\" (UID: \"ce5323ef-e7f2-4921-bed5-9e71e2aa835d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712e5ae1-s7d65" Nov 03 18:15:13.948690 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:13.948664 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5323ef-e7f2-4921-bed5-9e71e2aa835d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-712e5ae1-s7d65\" (UID: \"ce5323ef-e7f2-4921-bed5-9e71e2aa835d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712e5ae1-s7d65" Nov 03 18:15:13.949837 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:13.949817 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5323ef-e7f2-4921-bed5-9e71e2aa835d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-712e5ae1-s7d65\" (UID: \"ce5323ef-e7f2-4921-bed5-9e71e2aa835d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712e5ae1-s7d65" Nov 03 18:15:14.004853 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:14.004821 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n6dg\" (UniqueName: \"kubernetes.io/projected/ce5323ef-e7f2-4921-bed5-9e71e2aa835d-kube-api-access-7n6dg\") pod \"maas-api-revoke-sub-712e5ae1-s7d65\" (UID: \"ce5323ef-e7f2-4921-bed5-9e71e2aa835d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712e5ae1-s7d65" Nov 03 18:15:14.053005 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:14.052921 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:14.056445 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:14.056388 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-712e5ae1-s7d65" Nov 03 18:15:14.341130 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:14.341093 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:14.435851 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:14.435817 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-712e5ae1-s7d65"] Nov 03 18:15:14.441076 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:15:14.441050 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce5323ef_e7f2_4921_bed5_9e71e2aa835d.slice/crio-2bfa70a5d0113b89c7015f6b19ac31ac75c76a24803fb9fa872da3d555f8bd93 WatchSource:0}: Error finding container 2bfa70a5d0113b89c7015f6b19ac31ac75c76a24803fb9fa872da3d555f8bd93: Status 404 returned error can't find the container with id 2bfa70a5d0113b89c7015f6b19ac31ac75c76a24803fb9fa872da3d555f8bd93 Nov 03 18:15:14.490800 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:14.490767 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-712e5ae1-s7d65" event={"ID":"ce5323ef-e7f2-4921-bed5-9e71e2aa835d","Type":"ContainerStarted","Data":"2bfa70a5d0113b89c7015f6b19ac31ac75c76a24803fb9fa872da3d555f8bd93"} Nov 03 18:15:14.492309 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:14.492282 5805 generic.go:356] "Generic (PLEG): container finished" podID="00a224da-b74e-4186-b7ef-030cc5fc1c92" containerID="0692b73d3ecb0867617afc43dd9aebcfcae94523be347fe650282748e9cbe1a7" exitCode=0 Nov 03 18:15:14.492432 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:14.492363 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9525b615-wtwgt" event={"ID":"00a224da-b74e-4186-b7ef-030cc5fc1c92","Type":"ContainerDied","Data":"0692b73d3ecb0867617afc43dd9aebcfcae94523be347fe650282748e9cbe1a7"} Nov 03 18:15:14.542339 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:14.542314 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:14.790636 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:14.790603 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:14.977123 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:14.977035 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60483b95-19d2-46c3-ad70-5d8b36ffb572" path="/var/lib/kubelet/pods/60483b95-19d2-46c3-ad70-5d8b36ffb572/volumes" Nov 03 18:15:14.990498 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:14.990450 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:15.223173 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:15.223136 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:15.419178 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:15.419148 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:15.500825 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:15.500793 5805 generic.go:356] "Generic (PLEG): container finished" podID="ce5323ef-e7f2-4921-bed5-9e71e2aa835d" containerID="eec8d45ab6b3c2785110373653eaa9e371f7cfd603eac9722e703127e47f2f00" exitCode=0 Nov 03 18:15:15.500985 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:15.500878 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-712e5ae1-s7d65" event={"ID":"ce5323ef-e7f2-4921-bed5-9e71e2aa835d","Type":"ContainerDied","Data":"eec8d45ab6b3c2785110373653eaa9e371f7cfd603eac9722e703127e47f2f00"} Nov 03 18:15:15.636691 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:15.636663 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:15.647080 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:15.647025 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9525b615-wtwgt" Nov 03 18:15:15.680316 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:15.680281 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a224da-b74e-4186-b7ef-030cc5fc1c92-kube-api-access-ht8x6\" (UniqueName: \"kubernetes.io/projected/00a224da-b74e-4186-b7ef-030cc5fc1c92-kube-api-access-ht8x6\") pod \"00a224da-b74e-4186-b7ef-030cc5fc1c92\" (UID: \"00a224da-b74e-4186-b7ef-030cc5fc1c92\") " Nov 03 18:15:15.680510 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:15.680425 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00a224da-b74e-4186-b7ef-030cc5fc1c92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a224da-b74e-4186-b7ef-030cc5fc1c92-maas-api-xks-ca\") pod \"00a224da-b74e-4186-b7ef-030cc5fc1c92\" (UID: \"00a224da-b74e-4186-b7ef-030cc5fc1c92\") " Nov 03 18:15:15.680510 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:15.680473 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00a224da-b74e-4186-b7ef-030cc5fc1c92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a224da-b74e-4186-b7ef-030cc5fc1c92-maas-api-openshift-ca\") pod \"00a224da-b74e-4186-b7ef-030cc5fc1c92\" (UID: \"00a224da-b74e-4186-b7ef-030cc5fc1c92\") " Nov 03 18:15:15.680825 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:15.680801 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a224da-b74e-4186-b7ef-030cc5fc1c92-maas-api-openshift-ca" pod "00a224da-b74e-4186-b7ef-030cc5fc1c92" (UID: "00a224da-b74e-4186-b7ef-030cc5fc1c92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:15.681111 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:15.681081 5805 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00a224da-b74e-4186-b7ef-030cc5fc1c92-maas-api-openshift-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:15.682999 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:15.682968 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a224da-b74e-4186-b7ef-030cc5fc1c92-kube-api-access-ht8x6" pod "00a224da-b74e-4186-b7ef-030cc5fc1c92" (UID: "00a224da-b74e-4186-b7ef-030cc5fc1c92"). InnerVolumeSpecName "kube-api-access-ht8x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:15.683814 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:15.683787 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00a224da-b74e-4186-b7ef-030cc5fc1c92-maas-api-xks-ca" pod "00a224da-b74e-4186-b7ef-030cc5fc1c92" (UID: "00a224da-b74e-4186-b7ef-030cc5fc1c92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:15:15.782809 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:15.782778 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht8x6\" (UniqueName: \"kubernetes.io/projected/00a224da-b74e-4186-b7ef-030cc5fc1c92-kube-api-access-ht8x6\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:15.782809 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:15.782806 5805 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00a224da-b74e-4186-b7ef-030cc5fc1c92-maas-api-xks-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:15.790659 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:15.790624 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs"] Nov 03 18:15:15.790901 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:15.790873 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" podUID="5efde3b1-24ad-4da6-81e9-09c7c7aa181c" containerName="istio-proxy" containerID="cri-o://bf16ff46d54088f2855b6aeaf974c8d54195ae79252d3bd17cf036c8dc1596d9" gracePeriod=30 Nov 03 18:15:15.835757 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:15.835718 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:15.970216 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:15.970132 5805 scope.go:122] "RemoveContainer" containerID="6b663cc90cd3187c49fe1a0bc949033495802267cc0af99e8f52c04e4235f075" Nov 03 18:15:15.975572 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:15.975541 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd"] Nov 03 18:15:16.335414 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:16.335343 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x"] Nov 03 18:15:16.336026 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:16.335671 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" podUID="10efc538-5a7a-40ce-b6be-819a2cddbec3" containerName="istio-proxy" containerID="cri-o://3fecda1bdcbd492a83a22186e1898a033530916f0b4ba8f14f1ffe7003a3d4d2" gracePeriod=30 Nov 03 18:15:16.511239 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:16.511137 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9525b615-wtwgt" Nov 03 18:15:16.511239 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:16.511203 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9525b615-wtwgt" event={"ID":"00a224da-b74e-4186-b7ef-030cc5fc1c92","Type":"ContainerDied","Data":"cbf7a02b228255c08fa467106967b704f51eb2ea0f2317b604314ddd4c555231"} Nov 03 18:15:16.511856 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:16.511255 5805 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbf7a02b228255c08fa467106967b704f51eb2ea0f2317b604314ddd4c555231" Nov 03 18:15:16.516092 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:16.516061 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf" event={"ID":"d22875d5-06bf-4abe-9961-9f5c9922eac0","Type":"ContainerStarted","Data":"3471aec09511943241fdeb8036a2757d89e208288eca9dc1f45da1f9ad634034"} Nov 03 18:15:16.516453 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:16.516185 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" podUID="51204cd4-e74e-4d85-acf7-2118f81797d2" containerName="istio-proxy" containerID="cri-o://af6d367ce78fed85fc8d085cbb53215c8a3acfd13e1caf94be1885193cff1ec9" gracePeriod=30 Nov 03 18:15:16.748222 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:16.748198 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-712e5ae1-s7d65" Nov 03 18:15:16.803874 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:16.803805 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce5323ef-e7f2-4921-bed5-9e71e2aa835d-kube-api-access-7n6dg\" (UniqueName: \"kubernetes.io/projected/ce5323ef-e7f2-4921-bed5-9e71e2aa835d-kube-api-access-7n6dg\") pod \"ce5323ef-e7f2-4921-bed5-9e71e2aa835d\" (UID: \"ce5323ef-e7f2-4921-bed5-9e71e2aa835d\") " Nov 03 18:15:16.804005 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:16.803966 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce5323ef-e7f2-4921-bed5-9e71e2aa835d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5323ef-e7f2-4921-bed5-9e71e2aa835d-maas-api-openshift-ca\") pod \"ce5323ef-e7f2-4921-bed5-9e71e2aa835d\" (UID: \"ce5323ef-e7f2-4921-bed5-9e71e2aa835d\") " Nov 03 18:15:16.804079 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:16.804040 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce5323ef-e7f2-4921-bed5-9e71e2aa835d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5323ef-e7f2-4921-bed5-9e71e2aa835d-maas-api-xks-ca\") pod \"ce5323ef-e7f2-4921-bed5-9e71e2aa835d\" (UID: \"ce5323ef-e7f2-4921-bed5-9e71e2aa835d\") " Nov 03 18:15:16.805526 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:16.804806 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce5323ef-e7f2-4921-bed5-9e71e2aa835d-maas-api-openshift-ca" pod "ce5323ef-e7f2-4921-bed5-9e71e2aa835d" (UID: "ce5323ef-e7f2-4921-bed5-9e71e2aa835d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:16.806630 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:16.806602 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce5323ef-e7f2-4921-bed5-9e71e2aa835d-maas-api-xks-ca" pod "ce5323ef-e7f2-4921-bed5-9e71e2aa835d" (UID: "ce5323ef-e7f2-4921-bed5-9e71e2aa835d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:15:16.806851 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:16.806828 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce5323ef-e7f2-4921-bed5-9e71e2aa835d-kube-api-access-7n6dg" pod "ce5323ef-e7f2-4921-bed5-9e71e2aa835d" (UID: "ce5323ef-e7f2-4921-bed5-9e71e2aa835d"). InnerVolumeSpecName "kube-api-access-7n6dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:16.905300 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:16.905271 5805 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5323ef-e7f2-4921-bed5-9e71e2aa835d-maas-api-openshift-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:16.905300 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:16.905299 5805 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5323ef-e7f2-4921-bed5-9e71e2aa835d-maas-api-xks-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:16.905458 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:16.905310 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n6dg\" (UniqueName: \"kubernetes.io/projected/ce5323ef-e7f2-4921-bed5-9e71e2aa835d-kube-api-access-7n6dg\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:16.969373 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:16.969337 5805 scope.go:122] "RemoveContainer" containerID="737c519f22e8379fdbc889b6e34a6a3bdf45adaa84da840498282df894675f3b" Nov 03 18:15:16.969547 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:16.969457 5805 scope.go:122] "RemoveContainer" containerID="f9e53ef01d8bfd70d46d659eac381d98570c8f4e6dda241c0d68ff6788f425b4" Nov 03 18:15:16.969625 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:16.969599 5805 scope.go:122] "RemoveContainer" containerID="ed2d43f49db1aba0238a83b079fc844e18838c4615e38327de538e18cde47f64" Nov 03 18:15:17.528238 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:17.528156 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-712e5ae1-s7d65" Nov 03 18:15:17.528777 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:17.528156 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-712e5ae1-s7d65" event={"ID":"ce5323ef-e7f2-4921-bed5-9e71e2aa835d","Type":"ContainerDied","Data":"2bfa70a5d0113b89c7015f6b19ac31ac75c76a24803fb9fa872da3d555f8bd93"} Nov 03 18:15:17.528777 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:17.528294 5805 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bfa70a5d0113b89c7015f6b19ac31ac75c76a24803fb9fa872da3d555f8bd93" Nov 03 18:15:17.531974 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:17.531948 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v" event={"ID":"b78a7ba2-1042-492c-b52c-dca73afd27d5","Type":"ContainerStarted","Data":"6f87daf7b35e7a78ab367d0869f015f31dfe7b9d2d8e7818df5629f2b8470fd3"} Nov 03 18:15:17.534310 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:17.534286 5805 generic.go:356] "Generic (PLEG): container finished" podID="d22875d5-06bf-4abe-9961-9f5c9922eac0" containerID="3471aec09511943241fdeb8036a2757d89e208288eca9dc1f45da1f9ad634034" exitCode=60 Nov 03 18:15:17.534472 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:17.534364 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf" event={"ID":"d22875d5-06bf-4abe-9961-9f5c9922eac0","Type":"ContainerDied","Data":"3471aec09511943241fdeb8036a2757d89e208288eca9dc1f45da1f9ad634034"} Nov 03 18:15:17.534542 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:17.534482 5805 scope.go:122] "RemoveContainer" containerID="6b663cc90cd3187c49fe1a0bc949033495802267cc0af99e8f52c04e4235f075" Nov 03 18:15:17.534854 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:17.534833 5805 scope.go:122] "RemoveContainer" containerID="3471aec09511943241fdeb8036a2757d89e208288eca9dc1f45da1f9ad634034" Nov 03 18:15:17.535136 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:15:17.535111 5805 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf_odh-ai-gateway-infra(d22875d5-06bf-4abe-9961-9f5c9922eac0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf" podUID="d22875d5-06bf-4abe-9961-9f5c9922eac0" Nov 03 18:15:17.536530 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:17.536505 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz" event={"ID":"7221deb2-4d2f-4f43-b7a5-064464fc3991","Type":"ContainerStarted","Data":"ad4994a9190703669881258e80dcf3da10642f0cb7c9abc42cd7b7a3979f1f42"} Nov 03 18:15:17.538992 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:17.538970 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg" event={"ID":"73900785-3360-4c42-8a5d-8c0e258fc637","Type":"ContainerStarted","Data":"3f0aacec6243ad032e76060e715b6bece96115fd45f3896a47b9c8eea5679c17"} Nov 03 18:15:17.691501 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:17.691440 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v" podStartSLOduration=17.69142115 podStartE2EDuration="17.69142115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:17.688702752 +0000 UTC m=+1754.822331786" watchObservedRunningTime="2026-11-03 18:15:17.69142115 +0000 UTC m=+1754.825050157" Nov 03 18:15:17.799028 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:17.798927 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz" podStartSLOduration=17.798914246 podStartE2EDuration="17.798914246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:17.796092536 +0000 UTC m=+1754.929721559" watchObservedRunningTime="2026-11-03 18:15:17.798914246 +0000 UTC m=+1754.932543255" Nov 03 18:15:17.811490 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:17.811459 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf"] Nov 03 18:15:18.547069 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:18.547033 5805 generic.go:356] "Generic (PLEG): container finished" podID="b78a7ba2-1042-492c-b52c-dca73afd27d5" containerID="6f87daf7b35e7a78ab367d0869f015f31dfe7b9d2d8e7818df5629f2b8470fd3" exitCode=60 Nov 03 18:15:18.547543 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:18.547090 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v" event={"ID":"b78a7ba2-1042-492c-b52c-dca73afd27d5","Type":"ContainerDied","Data":"6f87daf7b35e7a78ab367d0869f015f31dfe7b9d2d8e7818df5629f2b8470fd3"} Nov 03 18:15:18.547543 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:18.547155 5805 scope.go:122] "RemoveContainer" containerID="ed2d43f49db1aba0238a83b079fc844e18838c4615e38327de538e18cde47f64" Nov 03 18:15:18.547543 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:18.547503 5805 scope.go:122] "RemoveContainer" containerID="6f87daf7b35e7a78ab367d0869f015f31dfe7b9d2d8e7818df5629f2b8470fd3" Nov 03 18:15:18.547824 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:15:18.547806 5805 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v_odh-ai-gateway-infra(b78a7ba2-1042-492c-b52c-dca73afd27d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v" podUID="b78a7ba2-1042-492c-b52c-dca73afd27d5" Nov 03 18:15:18.550964 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:18.550943 5805 generic.go:356] "Generic (PLEG): container finished" podID="7221deb2-4d2f-4f43-b7a5-064464fc3991" containerID="ad4994a9190703669881258e80dcf3da10642f0cb7c9abc42cd7b7a3979f1f42" exitCode=60 Nov 03 18:15:18.551055 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:18.551023 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz" event={"ID":"7221deb2-4d2f-4f43-b7a5-064464fc3991","Type":"ContainerDied","Data":"ad4994a9190703669881258e80dcf3da10642f0cb7c9abc42cd7b7a3979f1f42"} Nov 03 18:15:18.551490 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:18.551465 5805 scope.go:122] "RemoveContainer" containerID="ad4994a9190703669881258e80dcf3da10642f0cb7c9abc42cd7b7a3979f1f42" Nov 03 18:15:18.551827 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:15:18.551804 5805 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz_odh-ai-gateway-infra(7221deb2-4d2f-4f43-b7a5-064464fc3991)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz" podUID="7221deb2-4d2f-4f43-b7a5-064464fc3991" Nov 03 18:15:18.553007 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:18.552987 5805 generic.go:356] "Generic (PLEG): container finished" podID="73900785-3360-4c42-8a5d-8c0e258fc637" containerID="3f0aacec6243ad032e76060e715b6bece96115fd45f3896a47b9c8eea5679c17" exitCode=60 Nov 03 18:15:18.553077 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:18.553060 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg" event={"ID":"73900785-3360-4c42-8a5d-8c0e258fc637","Type":"ContainerDied","Data":"3f0aacec6243ad032e76060e715b6bece96115fd45f3896a47b9c8eea5679c17"} Nov 03 18:15:18.553324 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:18.553311 5805 scope.go:122] "RemoveContainer" containerID="3f0aacec6243ad032e76060e715b6bece96115fd45f3896a47b9c8eea5679c17" Nov 03 18:15:18.553738 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:15:18.553711 5805 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg_odh-ai-gateway-infra(73900785-3360-4c42-8a5d-8c0e258fc637)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg" podUID="73900785-3360-4c42-8a5d-8c0e258fc637" Nov 03 18:15:18.560246 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:18.560220 5805 scope.go:122] "RemoveContainer" containerID="f9e53ef01d8bfd70d46d659eac381d98570c8f4e6dda241c0d68ff6788f425b4" Nov 03 18:15:18.575094 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:18.575077 5805 scope.go:122] "RemoveContainer" containerID="737c519f22e8379fdbc889b6e34a6a3bdf45adaa84da840498282df894675f3b" Nov 03 18:15:18.609159 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:18.609122 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:15:18.609250 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:18.609184 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" podUID="10efc538-5a7a-40ce-b6be-819a2cddbec3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:15:18.701423 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:18.701380 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf" Nov 03 18:15:18.701688 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:18.701667 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v"] Nov 03 18:15:18.731913 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:18.731887 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d22875d5-06bf-4abe-9961-9f5c9922eac0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22875d5-06bf-4abe-9961-9f5c9922eac0-maas-api-service-ca\") pod \"d22875d5-06bf-4abe-9961-9f5c9922eac0\" (UID: \"d22875d5-06bf-4abe-9961-9f5c9922eac0\") " Nov 03 18:15:18.732025 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:18.731925 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d22875d5-06bf-4abe-9961-9f5c9922eac0-kube-api-access-p6v4s\" (UniqueName: \"kubernetes.io/projected/d22875d5-06bf-4abe-9961-9f5c9922eac0-kube-api-access-p6v4s\") pod \"d22875d5-06bf-4abe-9961-9f5c9922eac0\" (UID: \"d22875d5-06bf-4abe-9961-9f5c9922eac0\") " Nov 03 18:15:18.732225 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:18.732200 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d22875d5-06bf-4abe-9961-9f5c9922eac0-maas-api-service-ca" pod "d22875d5-06bf-4abe-9961-9f5c9922eac0" (UID: "d22875d5-06bf-4abe-9961-9f5c9922eac0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:18.734024 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:18.734002 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22875d5-06bf-4abe-9961-9f5c9922eac0-kube-api-access-p6v4s" pod "d22875d5-06bf-4abe-9961-9f5c9922eac0" (UID: "d22875d5-06bf-4abe-9961-9f5c9922eac0"). InnerVolumeSpecName "kube-api-access-p6v4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:18.807088 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:18.807065 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz"] Nov 03 18:15:18.833892 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:18.833866 5805 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d22875d5-06bf-4abe-9961-9f5c9922eac0-maas-api-service-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:18.833892 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:18.833892 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6v4s\" (UniqueName: \"kubernetes.io/projected/d22875d5-06bf-4abe-9961-9f5c9922eac0-kube-api-access-p6v4s\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:18.918017 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:18.917983 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg"] Nov 03 18:15:19.564257 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.564229 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf" Nov 03 18:15:19.564257 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.564236 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf" event={"ID":"d22875d5-06bf-4abe-9961-9f5c9922eac0","Type":"ContainerDied","Data":"8d5dc1687bd4cdaebe042ed047df02e404e110574c4c9a8eb7ac710c677973af"} Nov 03 18:15:19.564725 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.564302 5805 scope.go:122] "RemoveContainer" containerID="3471aec09511943241fdeb8036a2757d89e208288eca9dc1f45da1f9ad634034" Nov 03 18:15:19.748589 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.748458 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v" Nov 03 18:15:19.799291 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.798377 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:19.810335 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.810288 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz" Nov 03 18:15:19.849538 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.849509 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7221deb2-4d2f-4f43-b7a5-064464fc3991-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7221deb2-4d2f-4f43-b7a5-064464fc3991-maas-api-service-ca\") pod \"7221deb2-4d2f-4f43-b7a5-064464fc3991\" (UID: \"7221deb2-4d2f-4f43-b7a5-064464fc3991\") " Nov 03 18:15:19.849699 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.849682 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b78a7ba2-1042-492c-b52c-dca73afd27d5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78a7ba2-1042-492c-b52c-dca73afd27d5-maas-api-service-ca\") pod \"b78a7ba2-1042-492c-b52c-dca73afd27d5\" (UID: \"b78a7ba2-1042-492c-b52c-dca73afd27d5\") " Nov 03 18:15:19.849827 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.849808 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b78a7ba2-1042-492c-b52c-dca73afd27d5-kube-api-access-tqxzm\" (UniqueName: \"kubernetes.io/projected/b78a7ba2-1042-492c-b52c-dca73afd27d5-kube-api-access-tqxzm\") pod \"b78a7ba2-1042-492c-b52c-dca73afd27d5\" (UID: \"b78a7ba2-1042-492c-b52c-dca73afd27d5\") " Nov 03 18:15:19.849906 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.849850 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7221deb2-4d2f-4f43-b7a5-064464fc3991-kube-api-access-9qfbj\" (UniqueName: \"kubernetes.io/projected/7221deb2-4d2f-4f43-b7a5-064464fc3991-kube-api-access-9qfbj\") pod \"7221deb2-4d2f-4f43-b7a5-064464fc3991\" (UID: \"7221deb2-4d2f-4f43-b7a5-064464fc3991\") " Nov 03 18:15:19.849952 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.849911 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7221deb2-4d2f-4f43-b7a5-064464fc3991-maas-api-service-ca" pod "7221deb2-4d2f-4f43-b7a5-064464fc3991" (UID: "7221deb2-4d2f-4f43-b7a5-064464fc3991"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:19.850150 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.850122 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b78a7ba2-1042-492c-b52c-dca73afd27d5-maas-api-service-ca" pod "b78a7ba2-1042-492c-b52c-dca73afd27d5" (UID: "b78a7ba2-1042-492c-b52c-dca73afd27d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:19.850339 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.850324 5805 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7221deb2-4d2f-4f43-b7a5-064464fc3991-maas-api-service-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:19.850438 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.850350 5805 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78a7ba2-1042-492c-b52c-dca73afd27d5-maas-api-service-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:19.851933 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.851915 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7221deb2-4d2f-4f43-b7a5-064464fc3991-kube-api-access-9qfbj" pod "7221deb2-4d2f-4f43-b7a5-064464fc3991" (UID: "7221deb2-4d2f-4f43-b7a5-064464fc3991"). InnerVolumeSpecName "kube-api-access-9qfbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:19.852020 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.852010 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78a7ba2-1042-492c-b52c-dca73afd27d5-kube-api-access-tqxzm" pod "b78a7ba2-1042-492c-b52c-dca73afd27d5" (UID: "b78a7ba2-1042-492c-b52c-dca73afd27d5"). InnerVolumeSpecName "kube-api-access-tqxzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:19.857020 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.856999 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg" Nov 03 18:15:19.891526 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.891481 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf"] Nov 03 18:15:19.899723 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.899699 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-1720c00c-29895495-qnpkf"] Nov 03 18:15:19.950901 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.950873 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73900785-3360-4c42-8a5d-8c0e258fc637-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73900785-3360-4c42-8a5d-8c0e258fc637-maas-api-service-ca\") pod \"73900785-3360-4c42-8a5d-8c0e258fc637\" (UID: \"73900785-3360-4c42-8a5d-8c0e258fc637\") " Nov 03 18:15:19.951075 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.951062 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73900785-3360-4c42-8a5d-8c0e258fc637-kube-api-access-48jrd\" (UniqueName: \"kubernetes.io/projected/73900785-3360-4c42-8a5d-8c0e258fc637-kube-api-access-48jrd\") pod \"73900785-3360-4c42-8a5d-8c0e258fc637\" (UID: \"73900785-3360-4c42-8a5d-8c0e258fc637\") " Nov 03 18:15:19.951206 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.951184 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73900785-3360-4c42-8a5d-8c0e258fc637-maas-api-service-ca" pod "73900785-3360-4c42-8a5d-8c0e258fc637" (UID: "73900785-3360-4c42-8a5d-8c0e258fc637"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:19.951470 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.951453 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqxzm\" (UniqueName: \"kubernetes.io/projected/b78a7ba2-1042-492c-b52c-dca73afd27d5-kube-api-access-tqxzm\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:19.951509 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.951473 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qfbj\" (UniqueName: \"kubernetes.io/projected/7221deb2-4d2f-4f43-b7a5-064464fc3991-kube-api-access-9qfbj\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:19.951509 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.951483 5805 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73900785-3360-4c42-8a5d-8c0e258fc637-maas-api-service-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:19.952993 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:19.952974 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73900785-3360-4c42-8a5d-8c0e258fc637-kube-api-access-48jrd" pod "73900785-3360-4c42-8a5d-8c0e258fc637" (UID: "73900785-3360-4c42-8a5d-8c0e258fc637"). InnerVolumeSpecName "kube-api-access-48jrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:20.052379 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:20.052347 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48jrd\" (UniqueName: \"kubernetes.io/projected/73900785-3360-4c42-8a5d-8c0e258fc637-kube-api-access-48jrd\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:20.176733 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:20.176656 5805 patch_prober.go:29] interesting pod/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:15:20.176733 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:20.176705 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" podUID="5efde3b1-24ad-4da6-81e9-09c7c7aa181c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:15:20.576951 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:20.576921 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v" Nov 03 18:15:20.577498 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:20.576918 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v" event={"ID":"b78a7ba2-1042-492c-b52c-dca73afd27d5","Type":"ContainerDied","Data":"7f06148ea960aeb2d76307c4fe7fdd261ef3227a307dac4ec51f81a8d69ac03f"} Nov 03 18:15:20.577498 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:20.577084 5805 scope.go:122] "RemoveContainer" containerID="6f87daf7b35e7a78ab367d0869f015f31dfe7b9d2d8e7818df5629f2b8470fd3" Nov 03 18:15:20.579707 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:20.579686 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz" Nov 03 18:15:20.579838 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:20.579718 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz" event={"ID":"7221deb2-4d2f-4f43-b7a5-064464fc3991","Type":"ContainerDied","Data":"d3051418faae196b518e3886b454dabd0fbd5b8ff13aba6aec8ed3cf00ca5022"} Nov 03 18:15:20.581377 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:20.581343 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg" event={"ID":"73900785-3360-4c42-8a5d-8c0e258fc637","Type":"ContainerDied","Data":"58dae839baec8a297b9afdc93e21e788462ec3d3dc772ce94055b9ec3a14de12"} Nov 03 18:15:20.581505 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:20.581353 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg" Nov 03 18:15:20.587956 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:20.587938 5805 scope.go:122] "RemoveContainer" containerID="ad4994a9190703669881258e80dcf3da10642f0cb7c9abc42cd7b7a3979f1f42" Nov 03 18:15:20.598247 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:20.598231 5805 scope.go:122] "RemoveContainer" containerID="3f0aacec6243ad032e76060e715b6bece96115fd45f3896a47b9c8eea5679c17" Nov 03 18:15:20.734190 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:20.734162 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz"] Nov 03 18:15:20.738867 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:20.738843 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2b4f46-29895495-776mz"] Nov 03 18:15:20.840519 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:20.840456 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg"] Nov 03 18:15:20.844581 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:20.844557 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aab753-29895495-mtwhg"] Nov 03 18:15:20.902752 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:20.902728 5805 patch_prober.go:29] interesting pod/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:15:20.902869 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:20.902776 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" podUID="51204cd4-e74e-4d85-acf7-2118f81797d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:15:20.946096 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:20.946066 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v"] Nov 03 18:15:20.949934 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:20.949905 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-ef9633-29895495-ljp8v"] Nov 03 18:15:20.989918 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:20.989885 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7221deb2-4d2f-4f43-b7a5-064464fc3991" path="/var/lib/kubelet/pods/7221deb2-4d2f-4f43-b7a5-064464fc3991/volumes" Nov 03 18:15:20.990367 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:20.990351 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73900785-3360-4c42-8a5d-8c0e258fc637" path="/var/lib/kubelet/pods/73900785-3360-4c42-8a5d-8c0e258fc637/volumes" Nov 03 18:15:20.990884 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:20.990869 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b78a7ba2-1042-492c-b52c-dca73afd27d5" path="/var/lib/kubelet/pods/b78a7ba2-1042-492c-b52c-dca73afd27d5/volumes" Nov 03 18:15:20.991606 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:20.991590 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d22875d5-06bf-4abe-9961-9f5c9922eac0" path="/var/lib/kubelet/pods/d22875d5-06bf-4abe-9961-9f5c9922eac0/volumes" Nov 03 18:15:21.068191 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.068150 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:15:21.168026 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.167951 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-token\" (UniqueName: \"kubernetes.io/projected/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-token\") pod \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " Nov 03 18:15:21.168026 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.167995 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-podinfo\") pod \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " Nov 03 18:15:21.168246 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.168030 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-data\") pod \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " Nov 03 18:15:21.168246 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.168117 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-workload-certs\") pod \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " Nov 03 18:15:21.168246 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.168189 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-envoy\") pod \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " Nov 03 18:15:21.168246 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.168210 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-workload-socket\") pod \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " Nov 03 18:15:21.168461 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.168296 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-credential-socket\") pod \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " Nov 03 18:15:21.168461 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.168327 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istiod-ca-cert\") pod \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " Nov 03 18:15:21.168461 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.168414 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-kube-api-access-s27bw\" (UniqueName: \"kubernetes.io/projected/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-kube-api-access-s27bw\") pod \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\" (UID: \"5efde3b1-24ad-4da6-81e9-09c7c7aa181c\") " Nov 03 18:15:21.168461 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.168417 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-data" pod "5efde3b1-24ad-4da6-81e9-09c7c7aa181c" (UID: "5efde3b1-24ad-4da6-81e9-09c7c7aa181c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:21.168461 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.168450 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-workload-certs" pod "5efde3b1-24ad-4da6-81e9-09c7c7aa181c" (UID: "5efde3b1-24ad-4da6-81e9-09c7c7aa181c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:21.168727 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.168577 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-workload-socket" pod "5efde3b1-24ad-4da6-81e9-09c7c7aa181c" (UID: "5efde3b1-24ad-4da6-81e9-09c7c7aa181c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:21.168727 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.168592 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-credential-socket" pod "5efde3b1-24ad-4da6-81e9-09c7c7aa181c" (UID: "5efde3b1-24ad-4da6-81e9-09c7c7aa181c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:21.168813 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.168795 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istiod-ca-cert" pod "5efde3b1-24ad-4da6-81e9-09c7c7aa181c" (UID: "5efde3b1-24ad-4da6-81e9-09c7c7aa181c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:21.169065 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.169042 5805 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-data\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.169150 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.169070 5805 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-workload-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.169150 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.169104 5805 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-workload-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.169150 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.169119 5805 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-credential-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.169150 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.169134 5805 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istiod-ca-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.170381 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.170349 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-podinfo" pod "5efde3b1-24ad-4da6-81e9-09c7c7aa181c" (UID: "5efde3b1-24ad-4da6-81e9-09c7c7aa181c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:15:21.170691 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.170671 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-kube-api-access-s27bw" pod "5efde3b1-24ad-4da6-81e9-09c7c7aa181c" (UID: "5efde3b1-24ad-4da6-81e9-09c7c7aa181c"). InnerVolumeSpecName "kube-api-access-s27bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:21.170758 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.170714 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-envoy" pod "5efde3b1-24ad-4da6-81e9-09c7c7aa181c" (UID: "5efde3b1-24ad-4da6-81e9-09c7c7aa181c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:21.170758 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.170736 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-token" pod "5efde3b1-24ad-4da6-81e9-09c7c7aa181c" (UID: "5efde3b1-24ad-4da6-81e9-09c7c7aa181c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:21.270714 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.270680 5805 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-envoy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.270863 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.270720 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s27bw\" (UniqueName: \"kubernetes.io/projected/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-kube-api-access-s27bw\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.270863 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.270731 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.270863 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.270740 5805 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5efde3b1-24ad-4da6-81e9-09c7c7aa181c-istio-podinfo\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.489886 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.489845 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" podUID="277a1fec-5c82-449d-839f-e843fa0f813d" containerName="main" containerID="cri-o://06cae19bb410a9ea1655bb0e740511db4aba660ce4db15fe1b6889a3d00426fe" gracePeriod=45 Nov 03 18:15:21.539612 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.539556 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerName="main" containerID="cri-o://75bb4310cfc973dbeee45d21daa4f00884d45f80a820bb5f384c5389c8a2d1be" gracePeriod=45 Nov 03 18:15:21.588505 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.588469 5805 generic.go:356] "Generic (PLEG): container finished" podID="5efde3b1-24ad-4da6-81e9-09c7c7aa181c" containerID="bf16ff46d54088f2855b6aeaf974c8d54195ae79252d3bd17cf036c8dc1596d9" exitCode=0 Nov 03 18:15:21.588914 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.588534 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" Nov 03 18:15:21.588914 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.588543 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" event={"ID":"5efde3b1-24ad-4da6-81e9-09c7c7aa181c","Type":"ContainerDied","Data":"bf16ff46d54088f2855b6aeaf974c8d54195ae79252d3bd17cf036c8dc1596d9"} Nov 03 18:15:21.588914 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.588621 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs" event={"ID":"5efde3b1-24ad-4da6-81e9-09c7c7aa181c","Type":"ContainerDied","Data":"962f99a87724695a549e29cfae109ac2fde071c0b5b75b12467fef6dc7e975b4"} Nov 03 18:15:21.588914 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.588652 5805 scope.go:122] "RemoveContainer" containerID="bf16ff46d54088f2855b6aeaf974c8d54195ae79252d3bd17cf036c8dc1596d9" Nov 03 18:15:21.590354 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.590325 5805 generic.go:356] "Generic (PLEG): container finished" podID="10efc538-5a7a-40ce-b6be-819a2cddbec3" containerID="3fecda1bdcbd492a83a22186e1898a033530916f0b4ba8f14f1ffe7003a3d4d2" exitCode=0 Nov 03 18:15:21.590462 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.590382 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" event={"ID":"10efc538-5a7a-40ce-b6be-819a2cddbec3","Type":"ContainerDied","Data":"3fecda1bdcbd492a83a22186e1898a033530916f0b4ba8f14f1ffe7003a3d4d2"} Nov 03 18:15:21.657371 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.657338 5805 scope.go:122] "RemoveContainer" containerID="bf16ff46d54088f2855b6aeaf974c8d54195ae79252d3bd17cf036c8dc1596d9" Nov 03 18:15:21.657707 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:15:21.657678 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf16ff46d54088f2855b6aeaf974c8d54195ae79252d3bd17cf036c8dc1596d9\": container with ID starting with bf16ff46d54088f2855b6aeaf974c8d54195ae79252d3bd17cf036c8dc1596d9 not found: ID does not exist" containerID="bf16ff46d54088f2855b6aeaf974c8d54195ae79252d3bd17cf036c8dc1596d9" Nov 03 18:15:21.657813 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.657712 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf16ff46d54088f2855b6aeaf974c8d54195ae79252d3bd17cf036c8dc1596d9"} err="failed to get container status \"bf16ff46d54088f2855b6aeaf974c8d54195ae79252d3bd17cf036c8dc1596d9\": rpc error: code = NotFound desc = could not find container \"bf16ff46d54088f2855b6aeaf974c8d54195ae79252d3bd17cf036c8dc1596d9\": container with ID starting with bf16ff46d54088f2855b6aeaf974c8d54195ae79252d3bd17cf036c8dc1596d9 not found: ID does not exist" Nov 03 18:15:21.661897 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.661877 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:15:21.770794 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.769913 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs"] Nov 03 18:15:21.778607 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.777915 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-podinfo\") pod \"10efc538-5a7a-40ce-b6be-819a2cddbec3\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " Nov 03 18:15:21.778607 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.777980 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10efc538-5a7a-40ce-b6be-819a2cddbec3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10efc538-5a7a-40ce-b6be-819a2cddbec3-istiod-ca-cert\") pod \"10efc538-5a7a-40ce-b6be-819a2cddbec3\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " Nov 03 18:15:21.778607 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.778039 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-data\") pod \"10efc538-5a7a-40ce-b6be-819a2cddbec3\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " Nov 03 18:15:21.778607 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.778082 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10efc538-5a7a-40ce-b6be-819a2cddbec3-kube-api-access-wv5bp\" (UniqueName: \"kubernetes.io/projected/10efc538-5a7a-40ce-b6be-819a2cddbec3-kube-api-access-wv5bp\") pod \"10efc538-5a7a-40ce-b6be-819a2cddbec3\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " Nov 03 18:15:21.778607 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.778153 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-workload-certs\") pod \"10efc538-5a7a-40ce-b6be-819a2cddbec3\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " Nov 03 18:15:21.778607 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.778191 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-envoy\") pod \"10efc538-5a7a-40ce-b6be-819a2cddbec3\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " Nov 03 18:15:21.778607 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.778366 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-workload-socket\") pod \"10efc538-5a7a-40ce-b6be-819a2cddbec3\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " Nov 03 18:15:21.779849 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.779010 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-credential-socket\") pod \"10efc538-5a7a-40ce-b6be-819a2cddbec3\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " Nov 03 18:15:21.779849 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.779057 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-token\" (UniqueName: \"kubernetes.io/projected/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-token\") pod \"10efc538-5a7a-40ce-b6be-819a2cddbec3\" (UID: \"10efc538-5a7a-40ce-b6be-819a2cddbec3\") " Nov 03 18:15:21.781011 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.780671 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-workload-certs" pod "10efc538-5a7a-40ce-b6be-819a2cddbec3" (UID: "10efc538-5a7a-40ce-b6be-819a2cddbec3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:21.781011 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.780614 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10efc538-5a7a-40ce-b6be-819a2cddbec3-istiod-ca-cert" pod "10efc538-5a7a-40ce-b6be-819a2cddbec3" (UID: "10efc538-5a7a-40ce-b6be-819a2cddbec3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:21.781011 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.780886 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-credential-socket" pod "10efc538-5a7a-40ce-b6be-819a2cddbec3" (UID: "10efc538-5a7a-40ce-b6be-819a2cddbec3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:21.781011 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.780946 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-workload-socket" pod "10efc538-5a7a-40ce-b6be-819a2cddbec3" (UID: "10efc538-5a7a-40ce-b6be-819a2cddbec3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:21.781011 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.780992 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-data" pod "10efc538-5a7a-40ce-b6be-819a2cddbec3" (UID: "10efc538-5a7a-40ce-b6be-819a2cddbec3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:21.784813 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.784779 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-token" pod "10efc538-5a7a-40ce-b6be-819a2cddbec3" (UID: "10efc538-5a7a-40ce-b6be-819a2cddbec3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:21.786151 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.786118 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-envoy" pod "10efc538-5a7a-40ce-b6be-819a2cddbec3" (UID: "10efc538-5a7a-40ce-b6be-819a2cddbec3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:21.786151 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.786134 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10efc538-5a7a-40ce-b6be-819a2cddbec3-kube-api-access-wv5bp" pod "10efc538-5a7a-40ce-b6be-819a2cddbec3" (UID: "10efc538-5a7a-40ce-b6be-819a2cddbec3"). InnerVolumeSpecName "kube-api-access-wv5bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:21.786461 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.786436 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-podinfo" pod "10efc538-5a7a-40ce-b6be-819a2cddbec3" (UID: "10efc538-5a7a-40ce-b6be-819a2cddbec3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:15:21.843622 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.824249 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:21.847224 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.847183 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-pk5gs"] Nov 03 18:15:21.852658 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.852627 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:15:21.881760 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.880699 5805 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-workload-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.881760 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.880728 5805 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-credential-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.881760 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.880746 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.881760 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.880759 5805 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-podinfo\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.881760 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.880773 5805 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10efc538-5a7a-40ce-b6be-819a2cddbec3-istiod-ca-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.881760 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.880786 5805 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-data\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.881760 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.880800 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv5bp\" (UniqueName: \"kubernetes.io/projected/10efc538-5a7a-40ce-b6be-819a2cddbec3-kube-api-access-wv5bp\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.881760 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.880814 5805 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-workload-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.881760 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.880827 5805 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10efc538-5a7a-40ce-b6be-819a2cddbec3-istio-envoy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.883692 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.883620 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 18:15:21.886527 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.886509 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:15:21.981865 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.981786 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-workload-socket\") pod \"51204cd4-e74e-4d85-acf7-2118f81797d2\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " Nov 03 18:15:21.981865 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.981834 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-kube-api-access-vmrc9\" (UniqueName: \"kubernetes.io/projected/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-kube-api-access-vmrc9\") pod \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\" (UID: \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\") " Nov 03 18:15:21.981865 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.981856 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-tmp-dir\") pod \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\" (UID: \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\") " Nov 03 18:15:21.982145 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.981876 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51204cd4-e74e-4d85-acf7-2118f81797d2-kube-api-access-mq9mw\" (UniqueName: \"kubernetes.io/projected/51204cd4-e74e-4d85-acf7-2118f81797d2-kube-api-access-mq9mw\") pod \"51204cd4-e74e-4d85-acf7-2118f81797d2\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " Nov 03 18:15:21.982145 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.981930 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277a1fec-5c82-449d-839f-e843fa0f813d-kube-api-access-zvtdp\" (UniqueName: \"kubernetes.io/projected/277a1fec-5c82-449d-839f-e843fa0f813d-kube-api-access-zvtdp\") pod \"277a1fec-5c82-449d-839f-e843fa0f813d\" (UID: \"277a1fec-5c82-449d-839f-e843fa0f813d\") " Nov 03 18:15:21.982145 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.981947 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-podinfo\") pod \"51204cd4-e74e-4d85-acf7-2118f81797d2\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " Nov 03 18:15:21.982145 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.981966 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-home\" (UniqueName: \"kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-home\") pod \"277a1fec-5c82-449d-839f-e843fa0f813d\" (UID: \"277a1fec-5c82-449d-839f-e843fa0f813d\") " Nov 03 18:15:21.982145 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.981990 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-dshm\") pod \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\" (UID: \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\") " Nov 03 18:15:21.982145 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.982018 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51204cd4-e74e-4d85-acf7-2118f81797d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51204cd4-e74e-4d85-acf7-2118f81797d2-istiod-ca-cert\") pod \"51204cd4-e74e-4d85-acf7-2118f81797d2\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " Nov 03 18:15:21.982145 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.982058 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-data\") pod \"51204cd4-e74e-4d85-acf7-2118f81797d2\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " Nov 03 18:15:21.982145 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.982109 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-home\" (UniqueName: \"kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-home\") pod \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\" (UID: \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\") " Nov 03 18:15:21.982145 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.982113 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-workload-socket" pod "51204cd4-e74e-4d85-acf7-2118f81797d2" (UID: "51204cd4-e74e-4d85-acf7-2118f81797d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:21.982145 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.982117 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-tmp-dir" pod "82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" (UID: "82c4f5f3-b386-4ade-bfc9-d5de175e5bcc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:21.982718 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.982154 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-token\" (UniqueName: \"kubernetes.io/projected/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-token\") pod \"51204cd4-e74e-4d85-acf7-2118f81797d2\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " Nov 03 18:15:21.982718 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.982189 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-workload-certs\") pod \"51204cd4-e74e-4d85-acf7-2118f81797d2\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " Nov 03 18:15:21.982718 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.982214 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-tls-certs\" (UniqueName: \"kubernetes.io/secret/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-tls-certs\") pod \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\" (UID: \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\") " Nov 03 18:15:21.982718 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.982252 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-model-cache\") pod \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\" (UID: \"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc\") " Nov 03 18:15:21.982718 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.982292 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-dshm\") pod \"277a1fec-5c82-449d-839f-e843fa0f813d\" (UID: \"277a1fec-5c82-449d-839f-e843fa0f813d\") " Nov 03 18:15:21.982718 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.982366 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-data" pod "51204cd4-e74e-4d85-acf7-2118f81797d2" (UID: "51204cd4-e74e-4d85-acf7-2118f81797d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:21.982718 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.982425 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-model-cache\") pod \"277a1fec-5c82-449d-839f-e843fa0f813d\" (UID: \"277a1fec-5c82-449d-839f-e843fa0f813d\") " Nov 03 18:15:21.982718 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.982466 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-tmp-dir\") pod \"277a1fec-5c82-449d-839f-e843fa0f813d\" (UID: \"277a1fec-5c82-449d-839f-e843fa0f813d\") " Nov 03 18:15:21.982718 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.982523 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-home" pod "277a1fec-5c82-449d-839f-e843fa0f813d" (UID: "277a1fec-5c82-449d-839f-e843fa0f813d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:21.982718 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.982532 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/277a1fec-5c82-449d-839f-e843fa0f813d-tls-certs\" (UniqueName: \"kubernetes.io/secret/277a1fec-5c82-449d-839f-e843fa0f813d-tls-certs\") pod \"277a1fec-5c82-449d-839f-e843fa0f813d\" (UID: \"277a1fec-5c82-449d-839f-e843fa0f813d\") " Nov 03 18:15:21.982718 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.982597 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-credential-socket\") pod \"51204cd4-e74e-4d85-acf7-2118f81797d2\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " Nov 03 18:15:21.982718 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.982633 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-envoy\") pod \"51204cd4-e74e-4d85-acf7-2118f81797d2\" (UID: \"51204cd4-e74e-4d85-acf7-2118f81797d2\") " Nov 03 18:15:21.982718 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.982711 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51204cd4-e74e-4d85-acf7-2118f81797d2-istiod-ca-cert" pod "51204cd4-e74e-4d85-acf7-2118f81797d2" (UID: "51204cd4-e74e-4d85-acf7-2118f81797d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:21.984853 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.982732 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-model-cache" pod "82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" (UID: "82c4f5f3-b386-4ade-bfc9-d5de175e5bcc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:21.984853 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.982908 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-home" pod "82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" (UID: "82c4f5f3-b386-4ade-bfc9-d5de175e5bcc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:21.984853 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.983252 5805 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-workload-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.984853 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.983275 5805 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-tmp-dir\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.984853 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.983290 5805 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-home\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.984853 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.983303 5805 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51204cd4-e74e-4d85-acf7-2118f81797d2-istiod-ca-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.984853 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.983319 5805 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-data\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.984853 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.983338 5805 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-home\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.984853 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.983356 5805 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-model-cache\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.984853 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.983630 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-workload-certs" pod "51204cd4-e74e-4d85-acf7-2118f81797d2" (UID: "51204cd4-e74e-4d85-acf7-2118f81797d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:21.984853 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.983677 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-credential-socket" pod "51204cd4-e74e-4d85-acf7-2118f81797d2" (UID: "51204cd4-e74e-4d85-acf7-2118f81797d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:21.984853 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.983844 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-model-cache" pod "277a1fec-5c82-449d-839f-e843fa0f813d" (UID: "277a1fec-5c82-449d-839f-e843fa0f813d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:21.984853 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.984057 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-tmp-dir" pod "277a1fec-5c82-449d-839f-e843fa0f813d" (UID: "277a1fec-5c82-449d-839f-e843fa0f813d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:21.985480 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.984855 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-kube-api-access-vmrc9" pod "82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" (UID: "82c4f5f3-b386-4ade-bfc9-d5de175e5bcc"). InnerVolumeSpecName "kube-api-access-vmrc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:21.985480 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.985365 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-dshm" pod "82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" (UID: "82c4f5f3-b386-4ade-bfc9-d5de175e5bcc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:21.985558 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.985489 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277a1fec-5c82-449d-839f-e843fa0f813d-kube-api-access-zvtdp" pod "277a1fec-5c82-449d-839f-e843fa0f813d" (UID: "277a1fec-5c82-449d-839f-e843fa0f813d"). InnerVolumeSpecName "kube-api-access-zvtdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:21.985558 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.985550 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-podinfo" pod "51204cd4-e74e-4d85-acf7-2118f81797d2" (UID: "51204cd4-e74e-4d85-acf7-2118f81797d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:15:21.985914 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.985891 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-dshm" pod "277a1fec-5c82-449d-839f-e843fa0f813d" (UID: "277a1fec-5c82-449d-839f-e843fa0f813d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:21.986199 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.986178 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-tls-certs" pod "82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" (UID: "82c4f5f3-b386-4ade-bfc9-d5de175e5bcc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:15:21.986812 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.986793 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51204cd4-e74e-4d85-acf7-2118f81797d2-kube-api-access-mq9mw" pod "51204cd4-e74e-4d85-acf7-2118f81797d2" (UID: "51204cd4-e74e-4d85-acf7-2118f81797d2"). InnerVolumeSpecName "kube-api-access-mq9mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:21.987026 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.987007 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-token" pod "51204cd4-e74e-4d85-acf7-2118f81797d2" (UID: "51204cd4-e74e-4d85-acf7-2118f81797d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:21.987112 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.987078 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-envoy" pod "51204cd4-e74e-4d85-acf7-2118f81797d2" (UID: "51204cd4-e74e-4d85-acf7-2118f81797d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:21.987233 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:21.987220 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/277a1fec-5c82-449d-839f-e843fa0f813d-tls-certs" pod "277a1fec-5c82-449d-839f-e843fa0f813d" (UID: "277a1fec-5c82-449d-839f-e843fa0f813d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:15:22.059504 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.059383 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:22.086747 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.086503 5805 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-dshm\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:22.086747 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.086534 5805 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-model-cache\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:22.086747 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.086550 5805 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/277a1fec-5c82-449d-839f-e843fa0f813d-tmp-dir\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:22.086747 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.086564 5805 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/277a1fec-5c82-449d-839f-e843fa0f813d-tls-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:22.086747 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.086577 5805 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-credential-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:22.086747 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.086592 5805 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-envoy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:22.086747 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.086606 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmrc9\" (UniqueName: \"kubernetes.io/projected/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-kube-api-access-vmrc9\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:22.086747 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.086621 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq9mw\" (UniqueName: \"kubernetes.io/projected/51204cd4-e74e-4d85-acf7-2118f81797d2-kube-api-access-mq9mw\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:22.086747 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.086634 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvtdp\" (UniqueName: \"kubernetes.io/projected/277a1fec-5c82-449d-839f-e843fa0f813d-kube-api-access-zvtdp\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:22.086747 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.086649 5805 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-podinfo\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:22.086747 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.086661 5805 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-dshm\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:22.086747 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.086674 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51204cd4-e74e-4d85-acf7-2118f81797d2-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:22.086747 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.086687 5805 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51204cd4-e74e-4d85-acf7-2118f81797d2-workload-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:22.086747 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.086700 5805 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc-tls-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:22.092389 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.090929 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7"] Nov 03 18:15:22.092389 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.091302 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" podUID="7d997ac9-3961-424f-88c5-7b2c2168eab9" containerName="istio-proxy" containerID="cri-o://7da719c73c98f1e01ec91b213feac532001b38059a0dd10d3c91d20e1c5ea843" gracePeriod=30 Nov 03 18:15:22.097505 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.095859 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v"] Nov 03 18:15:22.097505 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.096093 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" podUID="f403b27f-184b-42fd-839a-92fa3b4cdc2d" containerName="istio-proxy" containerID="cri-o://a1ef37a20a86aeb6b14d713c064c54d9c5e4b4dd435cf111e33341c312ecce7c" gracePeriod=30 Nov 03 18:15:22.100355 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.100332 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw"] Nov 03 18:15:22.100758 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.100722 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" podUID="7a819308-670a-4a83-8512-630337c61d5a" containerName="istio-proxy" containerID="cri-o://c6b9756f2536b09d7a58f7c3b85de440249c2f523e8e2bcab3a53fed2f27e604" gracePeriod=30 Nov 03 18:15:22.301426 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.301275 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:22.516907 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.516872 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:22.600962 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.600930 5805 generic.go:356] "Generic (PLEG): container finished" podID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerID="75bb4310cfc973dbeee45d21daa4f00884d45f80a820bb5f384c5389c8a2d1be" exitCode=0 Nov 03 18:15:22.601424 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.600976 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" Nov 03 18:15:22.601424 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.601011 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" event={"ID":"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc","Type":"ContainerDied","Data":"75bb4310cfc973dbeee45d21daa4f00884d45f80a820bb5f384c5389c8a2d1be"} Nov 03 18:15:22.601424 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.601047 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl" event={"ID":"82c4f5f3-b386-4ade-bfc9-d5de175e5bcc","Type":"ContainerDied","Data":"dd92b896d60ff47f9570d0c34154028cfc130443aa534cd183324360dd343706"} Nov 03 18:15:22.601424 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.601065 5805 scope.go:122] "RemoveContainer" containerID="75bb4310cfc973dbeee45d21daa4f00884d45f80a820bb5f384c5389c8a2d1be" Nov 03 18:15:22.602588 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.602555 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" Nov 03 18:15:22.602588 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.602571 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x" event={"ID":"10efc538-5a7a-40ce-b6be-819a2cddbec3","Type":"ContainerDied","Data":"cd96d2f32030bb7fada9156a1b8e75be6e837345fb4aee762812204a3b70eb30"} Nov 03 18:15:22.604168 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.604141 5805 generic.go:356] "Generic (PLEG): container finished" podID="277a1fec-5c82-449d-839f-e843fa0f813d" containerID="06cae19bb410a9ea1655bb0e740511db4aba660ce4db15fe1b6889a3d00426fe" exitCode=0 Nov 03 18:15:22.604269 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.604170 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" Nov 03 18:15:22.604269 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.604215 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" event={"ID":"277a1fec-5c82-449d-839f-e843fa0f813d","Type":"ContainerDied","Data":"06cae19bb410a9ea1655bb0e740511db4aba660ce4db15fe1b6889a3d00426fe"} Nov 03 18:15:22.604269 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.604247 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn" event={"ID":"277a1fec-5c82-449d-839f-e843fa0f813d","Type":"ContainerDied","Data":"98df22d70fb79b12264bc4e84a3c46e64dd1d42507a03e4575d7cfc4b87ade14"} Nov 03 18:15:22.617268 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.605863 5805 generic.go:356] "Generic (PLEG): container finished" podID="51204cd4-e74e-4d85-acf7-2118f81797d2" containerID="af6d367ce78fed85fc8d085cbb53215c8a3acfd13e1caf94be1885193cff1ec9" exitCode=0 Nov 03 18:15:22.617268 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.606042 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" Nov 03 18:15:22.617268 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.606072 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" event={"ID":"51204cd4-e74e-4d85-acf7-2118f81797d2","Type":"ContainerDied","Data":"af6d367ce78fed85fc8d085cbb53215c8a3acfd13e1caf94be1885193cff1ec9"} Nov 03 18:15:22.617268 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.606114 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd" event={"ID":"51204cd4-e74e-4d85-acf7-2118f81797d2","Type":"ContainerDied","Data":"b4cea092ddf8cddaa811459da523b730a2a8f7372be42d0b44c2c7679c347d60"} Nov 03 18:15:22.623662 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.623644 5805 scope.go:122] "RemoveContainer" containerID="75bb4310cfc973dbeee45d21daa4f00884d45f80a820bb5f384c5389c8a2d1be" Nov 03 18:15:22.624003 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:15:22.623975 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75bb4310cfc973dbeee45d21daa4f00884d45f80a820bb5f384c5389c8a2d1be\": container with ID starting with 75bb4310cfc973dbeee45d21daa4f00884d45f80a820bb5f384c5389c8a2d1be not found: ID does not exist" containerID="75bb4310cfc973dbeee45d21daa4f00884d45f80a820bb5f384c5389c8a2d1be" Nov 03 18:15:22.624178 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.624137 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75bb4310cfc973dbeee45d21daa4f00884d45f80a820bb5f384c5389c8a2d1be"} err="failed to get container status \"75bb4310cfc973dbeee45d21daa4f00884d45f80a820bb5f384c5389c8a2d1be\": rpc error: code = NotFound desc = could not find container \"75bb4310cfc973dbeee45d21daa4f00884d45f80a820bb5f384c5389c8a2d1be\": container with ID starting with 75bb4310cfc973dbeee45d21daa4f00884d45f80a820bb5f384c5389c8a2d1be not found: ID does not exist" Nov 03 18:15:22.624325 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.624179 5805 scope.go:122] "RemoveContainer" containerID="3fecda1bdcbd492a83a22186e1898a033530916f0b4ba8f14f1ffe7003a3d4d2" Nov 03 18:15:22.639244 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.639220 5805 scope.go:122] "RemoveContainer" containerID="06cae19bb410a9ea1655bb0e740511db4aba660ce4db15fe1b6889a3d00426fe" Nov 03 18:15:22.651387 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.651365 5805 scope.go:122] "RemoveContainer" containerID="06cae19bb410a9ea1655bb0e740511db4aba660ce4db15fe1b6889a3d00426fe" Nov 03 18:15:22.651879 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:15:22.651856 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06cae19bb410a9ea1655bb0e740511db4aba660ce4db15fe1b6889a3d00426fe\": container with ID starting with 06cae19bb410a9ea1655bb0e740511db4aba660ce4db15fe1b6889a3d00426fe not found: ID does not exist" containerID="06cae19bb410a9ea1655bb0e740511db4aba660ce4db15fe1b6889a3d00426fe" Nov 03 18:15:22.651977 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.651888 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06cae19bb410a9ea1655bb0e740511db4aba660ce4db15fe1b6889a3d00426fe"} err="failed to get container status \"06cae19bb410a9ea1655bb0e740511db4aba660ce4db15fe1b6889a3d00426fe\": rpc error: code = NotFound desc = could not find container \"06cae19bb410a9ea1655bb0e740511db4aba660ce4db15fe1b6889a3d00426fe\": container with ID starting with 06cae19bb410a9ea1655bb0e740511db4aba660ce4db15fe1b6889a3d00426fe not found: ID does not exist" Nov 03 18:15:22.651977 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.651907 5805 scope.go:122] "RemoveContainer" containerID="af6d367ce78fed85fc8d085cbb53215c8a3acfd13e1caf94be1885193cff1ec9" Nov 03 18:15:22.660968 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.660948 5805 scope.go:122] "RemoveContainer" containerID="af6d367ce78fed85fc8d085cbb53215c8a3acfd13e1caf94be1885193cff1ec9" Nov 03 18:15:22.661247 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:15:22.661228 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af6d367ce78fed85fc8d085cbb53215c8a3acfd13e1caf94be1885193cff1ec9\": container with ID starting with af6d367ce78fed85fc8d085cbb53215c8a3acfd13e1caf94be1885193cff1ec9 not found: ID does not exist" containerID="af6d367ce78fed85fc8d085cbb53215c8a3acfd13e1caf94be1885193cff1ec9" Nov 03 18:15:22.661296 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.661254 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af6d367ce78fed85fc8d085cbb53215c8a3acfd13e1caf94be1885193cff1ec9"} err="failed to get container status \"af6d367ce78fed85fc8d085cbb53215c8a3acfd13e1caf94be1885193cff1ec9\": rpc error: code = NotFound desc = could not find container \"af6d367ce78fed85fc8d085cbb53215c8a3acfd13e1caf94be1885193cff1ec9\": container with ID starting with af6d367ce78fed85fc8d085cbb53215c8a3acfd13e1caf94be1885193cff1ec9 not found: ID does not exist" Nov 03 18:15:22.726072 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.726042 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:22.754804 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.754776 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl"] Nov 03 18:15:22.759208 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.759179 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-a86228/simulator-w4-a86228-kserve-6d6685d47d-5pcvl"] Nov 03 18:15:22.863631 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.863556 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x"] Nov 03 18:15:22.870680 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.870651 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-4p42x"] Nov 03 18:15:22.977540 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.977510 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10efc538-5a7a-40ce-b6be-819a2cddbec3" path="/var/lib/kubelet/pods/10efc538-5a7a-40ce-b6be-819a2cddbec3/volumes" Nov 03 18:15:22.978113 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.978094 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5efde3b1-24ad-4da6-81e9-09c7c7aa181c" path="/var/lib/kubelet/pods/5efde3b1-24ad-4da6-81e9-09c7c7aa181c/volumes" Nov 03 18:15:22.978528 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.978514 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" path="/var/lib/kubelet/pods/82c4f5f3-b386-4ade-bfc9-d5de175e5bcc/volumes" Nov 03 18:15:22.979112 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.979086 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd"] Nov 03 18:15:22.980019 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:22.979997 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-k5xpd"] Nov 03 18:15:23.082011 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:23.081980 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn"] Nov 03 18:15:23.086200 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:23.086175 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-a86228/distinct-2-w4-a86228-kserve-5988444c97-9nqsn"] Nov 03 18:15:24.467758 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:24.467723 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:24.976662 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:24.976629 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="277a1fec-5c82-449d-839f-e843fa0f813d" path="/var/lib/kubelet/pods/277a1fec-5c82-449d-839f-e843fa0f813d/volumes" Nov 03 18:15:24.977113 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:24.977099 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51204cd4-e74e-4d85-acf7-2118f81797d2" path="/var/lib/kubelet/pods/51204cd4-e74e-4d85-acf7-2118f81797d2/volumes" Nov 03 18:15:26.217673 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:26.217637 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:15:26.218050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:26.217698 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" podUID="7a819308-670a-4a83-8512-630337c61d5a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:15:26.236165 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:15:26.236131 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:15:26.328463 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:26.328428 5805 patch_prober.go:29] interesting pod/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:15:26.328612 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:26.328481 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" podUID="7d997ac9-3961-424f-88c5-7b2c2168eab9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:15:27.292251 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.292228 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:15:27.364677 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.364648 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-envoy\") pod \"7d997ac9-3961-424f-88c5-7b2c2168eab9\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " Nov 03 18:15:27.364820 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.364711 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-podinfo\") pod \"7d997ac9-3961-424f-88c5-7b2c2168eab9\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " Nov 03 18:15:27.364820 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.364746 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-workload-socket\") pod \"7d997ac9-3961-424f-88c5-7b2c2168eab9\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " Nov 03 18:15:27.364930 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.364839 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d997ac9-3961-424f-88c5-7b2c2168eab9-kube-api-access-6xznk\" (UniqueName: \"kubernetes.io/projected/7d997ac9-3961-424f-88c5-7b2c2168eab9-kube-api-access-6xznk\") pod \"7d997ac9-3961-424f-88c5-7b2c2168eab9\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " Nov 03 18:15:27.364930 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.364875 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-workload-certs\") pod \"7d997ac9-3961-424f-88c5-7b2c2168eab9\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " Nov 03 18:15:27.364930 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.364911 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-credential-socket\") pod \"7d997ac9-3961-424f-88c5-7b2c2168eab9\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " Nov 03 18:15:27.365086 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.365002 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-token\" (UniqueName: \"kubernetes.io/projected/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-token\") pod \"7d997ac9-3961-424f-88c5-7b2c2168eab9\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " Nov 03 18:15:27.365086 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.365039 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d997ac9-3961-424f-88c5-7b2c2168eab9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d997ac9-3961-424f-88c5-7b2c2168eab9-istiod-ca-cert\") pod \"7d997ac9-3961-424f-88c5-7b2c2168eab9\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " Nov 03 18:15:27.365086 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.365074 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-data\") pod \"7d997ac9-3961-424f-88c5-7b2c2168eab9\" (UID: \"7d997ac9-3961-424f-88c5-7b2c2168eab9\") " Nov 03 18:15:27.365241 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.365159 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-workload-socket" pod "7d997ac9-3961-424f-88c5-7b2c2168eab9" (UID: "7d997ac9-3961-424f-88c5-7b2c2168eab9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:27.365241 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.365170 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-workload-certs" pod "7d997ac9-3961-424f-88c5-7b2c2168eab9" (UID: "7d997ac9-3961-424f-88c5-7b2c2168eab9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:27.365343 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.365246 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-credential-socket" pod "7d997ac9-3961-424f-88c5-7b2c2168eab9" (UID: "7d997ac9-3961-424f-88c5-7b2c2168eab9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:27.365560 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.365516 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-data" pod "7d997ac9-3961-424f-88c5-7b2c2168eab9" (UID: "7d997ac9-3961-424f-88c5-7b2c2168eab9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:27.365560 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.365528 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d997ac9-3961-424f-88c5-7b2c2168eab9-istiod-ca-cert" pod "7d997ac9-3961-424f-88c5-7b2c2168eab9" (UID: "7d997ac9-3961-424f-88c5-7b2c2168eab9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:27.365735 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.365678 5805 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d997ac9-3961-424f-88c5-7b2c2168eab9-istiod-ca-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:27.365735 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.365699 5805 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-data\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:27.365735 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.365713 5805 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-workload-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:27.365735 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.365730 5805 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-workload-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:27.365974 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.365745 5805 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-credential-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:27.366997 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.366971 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-envoy" pod "7d997ac9-3961-424f-88c5-7b2c2168eab9" (UID: "7d997ac9-3961-424f-88c5-7b2c2168eab9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:27.367105 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.367075 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-podinfo" pod "7d997ac9-3961-424f-88c5-7b2c2168eab9" (UID: "7d997ac9-3961-424f-88c5-7b2c2168eab9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:15:27.367166 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.367108 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d997ac9-3961-424f-88c5-7b2c2168eab9-kube-api-access-6xznk" pod "7d997ac9-3961-424f-88c5-7b2c2168eab9" (UID: "7d997ac9-3961-424f-88c5-7b2c2168eab9"). InnerVolumeSpecName "kube-api-access-6xznk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:27.367166 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.367135 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-token" pod "7d997ac9-3961-424f-88c5-7b2c2168eab9" (UID: "7d997ac9-3961-424f-88c5-7b2c2168eab9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:27.467183 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.467144 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xznk\" (UniqueName: \"kubernetes.io/projected/7d997ac9-3961-424f-88c5-7b2c2168eab9-kube-api-access-6xznk\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:27.467183 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.467173 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:27.467183 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.467183 5805 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-envoy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:27.467183 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.467191 5805 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d997ac9-3961-424f-88c5-7b2c2168eab9-istio-podinfo\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:27.669115 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.669077 5805 generic.go:356] "Generic (PLEG): container finished" podID="7a819308-670a-4a83-8512-630337c61d5a" containerID="c6b9756f2536b09d7a58f7c3b85de440249c2f523e8e2bcab3a53fed2f27e604" exitCode=0 Nov 03 18:15:27.669387 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.669366 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" event={"ID":"7a819308-670a-4a83-8512-630337c61d5a","Type":"ContainerDied","Data":"c6b9756f2536b09d7a58f7c3b85de440249c2f523e8e2bcab3a53fed2f27e604"} Nov 03 18:15:27.672643 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.672529 5805 generic.go:356] "Generic (PLEG): container finished" podID="f403b27f-184b-42fd-839a-92fa3b4cdc2d" containerID="a1ef37a20a86aeb6b14d713c064c54d9c5e4b4dd435cf111e33341c312ecce7c" exitCode=0 Nov 03 18:15:27.672643 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.672604 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" event={"ID":"f403b27f-184b-42fd-839a-92fa3b4cdc2d","Type":"ContainerDied","Data":"a1ef37a20a86aeb6b14d713c064c54d9c5e4b4dd435cf111e33341c312ecce7c"} Nov 03 18:15:27.674439 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.674192 5805 generic.go:356] "Generic (PLEG): container finished" podID="7d997ac9-3961-424f-88c5-7b2c2168eab9" containerID="7da719c73c98f1e01ec91b213feac532001b38059a0dd10d3c91d20e1c5ea843" exitCode=0 Nov 03 18:15:27.674439 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.674248 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" Nov 03 18:15:27.674439 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.674283 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" event={"ID":"7d997ac9-3961-424f-88c5-7b2c2168eab9","Type":"ContainerDied","Data":"7da719c73c98f1e01ec91b213feac532001b38059a0dd10d3c91d20e1c5ea843"} Nov 03 18:15:27.674439 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.674315 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7" event={"ID":"7d997ac9-3961-424f-88c5-7b2c2168eab9","Type":"ContainerDied","Data":"d9cdb0f5e663b66f1632256724960d74c1213abc2d7525c1c57f7d39839f0fa1"} Nov 03 18:15:27.674439 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.674335 5805 scope.go:122] "RemoveContainer" containerID="7da719c73c98f1e01ec91b213feac532001b38059a0dd10d3c91d20e1c5ea843" Nov 03 18:15:27.687739 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.687720 5805 scope.go:122] "RemoveContainer" containerID="7da719c73c98f1e01ec91b213feac532001b38059a0dd10d3c91d20e1c5ea843" Nov 03 18:15:27.688022 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:15:27.687999 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da719c73c98f1e01ec91b213feac532001b38059a0dd10d3c91d20e1c5ea843\": container with ID starting with 7da719c73c98f1e01ec91b213feac532001b38059a0dd10d3c91d20e1c5ea843 not found: ID does not exist" containerID="7da719c73c98f1e01ec91b213feac532001b38059a0dd10d3c91d20e1c5ea843" Nov 03 18:15:27.688110 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.688033 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da719c73c98f1e01ec91b213feac532001b38059a0dd10d3c91d20e1c5ea843"} err="failed to get container status \"7da719c73c98f1e01ec91b213feac532001b38059a0dd10d3c91d20e1c5ea843\": rpc error: code = NotFound desc = could not find container \"7da719c73c98f1e01ec91b213feac532001b38059a0dd10d3c91d20e1c5ea843\": container with ID starting with 7da719c73c98f1e01ec91b213feac532001b38059a0dd10d3c91d20e1c5ea843 not found: ID does not exist" Nov 03 18:15:27.802632 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.802604 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:15:27.813277 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.813251 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:15:27.832345 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.832321 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7"] Nov 03 18:15:27.836329 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.836306 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-4lhv7"] Nov 03 18:15:27.974122 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.974044 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a819308-670a-4a83-8512-630337c61d5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a819308-670a-4a83-8512-630337c61d5a-istio-podinfo\") pod \"7a819308-670a-4a83-8512-630337c61d5a\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " Nov 03 18:15:27.974122 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.974098 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a819308-670a-4a83-8512-630337c61d5a-kube-api-access-vsl85\" (UniqueName: \"kubernetes.io/projected/7a819308-670a-4a83-8512-630337c61d5a-kube-api-access-vsl85\") pod \"7a819308-670a-4a83-8512-630337c61d5a\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " Nov 03 18:15:27.974359 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.974215 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-credential-socket\") pod \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " Nov 03 18:15:27.974359 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.974278 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-workload-certs\") pod \"7a819308-670a-4a83-8512-630337c61d5a\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " Nov 03 18:15:27.974359 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.974351 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-workload-socket\") pod \"7a819308-670a-4a83-8512-630337c61d5a\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " Nov 03 18:15:27.974546 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.974389 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-istio-data\") pod \"7a819308-670a-4a83-8512-630337c61d5a\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " Nov 03 18:15:27.974546 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.974444 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-token\" (UniqueName: \"kubernetes.io/projected/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-token\") pod \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " Nov 03 18:15:27.974546 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.974484 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-istio-envoy\") pod \"7a819308-670a-4a83-8512-630337c61d5a\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " Nov 03 18:15:27.974546 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.974507 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-credential-socket" pod "f403b27f-184b-42fd-839a-92fa3b4cdc2d" (UID: "f403b27f-184b-42fd-839a-92fa3b4cdc2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:27.974684 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.974611 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-workload-certs" pod "7a819308-670a-4a83-8512-630337c61d5a" (UID: "7a819308-670a-4a83-8512-630337c61d5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:27.974684 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.974629 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-envoy\") pod \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " Nov 03 18:15:27.974746 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.974697 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-podinfo\") pod \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " Nov 03 18:15:27.974746 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.974703 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-workload-socket" pod "7a819308-670a-4a83-8512-630337c61d5a" (UID: "7a819308-670a-4a83-8512-630337c61d5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:27.974746 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.974727 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-credential-socket\") pod \"7a819308-670a-4a83-8512-630337c61d5a\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " Nov 03 18:15:27.974838 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.974784 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-data\") pod \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " Nov 03 18:15:27.974838 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.974821 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-workload-certs\") pod \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " Nov 03 18:15:27.974902 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.974849 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istiod-ca-cert\") pod \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " Nov 03 18:15:27.974902 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.974880 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f403b27f-184b-42fd-839a-92fa3b4cdc2d-kube-api-access-r8rl6\" (UniqueName: \"kubernetes.io/projected/f403b27f-184b-42fd-839a-92fa3b4cdc2d-kube-api-access-r8rl6\") pod \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " Nov 03 18:15:27.975008 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.974907 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-workload-socket\") pod \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\" (UID: \"f403b27f-184b-42fd-839a-92fa3b4cdc2d\") " Nov 03 18:15:27.975008 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.974942 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a819308-670a-4a83-8512-630337c61d5a-istio-token\" (UniqueName: \"kubernetes.io/projected/7a819308-670a-4a83-8512-630337c61d5a-istio-token\") pod \"7a819308-670a-4a83-8512-630337c61d5a\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " Nov 03 18:15:27.975008 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.974969 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a819308-670a-4a83-8512-630337c61d5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a819308-670a-4a83-8512-630337c61d5a-istiod-ca-cert\") pod \"7a819308-670a-4a83-8512-630337c61d5a\" (UID: \"7a819308-670a-4a83-8512-630337c61d5a\") " Nov 03 18:15:27.975339 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.975181 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-istio-data" pod "7a819308-670a-4a83-8512-630337c61d5a" (UID: "7a819308-670a-4a83-8512-630337c61d5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:27.975831 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.975558 5805 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-credential-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:27.975831 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.975583 5805 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-workload-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:27.975831 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.975598 5805 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-workload-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:27.975831 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.975604 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istiod-ca-cert" pod "f403b27f-184b-42fd-839a-92fa3b4cdc2d" (UID: "f403b27f-184b-42fd-839a-92fa3b4cdc2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:27.975831 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.975613 5805 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-istio-data\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:27.976687 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.976173 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-data" pod "f403b27f-184b-42fd-839a-92fa3b4cdc2d" (UID: "f403b27f-184b-42fd-839a-92fa3b4cdc2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:27.976687 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.976347 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-credential-socket" pod "7a819308-670a-4a83-8512-630337c61d5a" (UID: "7a819308-670a-4a83-8512-630337c61d5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:27.976687 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.976350 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-workload-certs" pod "f403b27f-184b-42fd-839a-92fa3b4cdc2d" (UID: "f403b27f-184b-42fd-839a-92fa3b4cdc2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:27.976687 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.976575 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-workload-socket" pod "f403b27f-184b-42fd-839a-92fa3b4cdc2d" (UID: "f403b27f-184b-42fd-839a-92fa3b4cdc2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:27.976687 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.976655 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a819308-670a-4a83-8512-630337c61d5a-istiod-ca-cert" pod "7a819308-670a-4a83-8512-630337c61d5a" (UID: "7a819308-670a-4a83-8512-630337c61d5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:27.978648 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.978607 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f403b27f-184b-42fd-839a-92fa3b4cdc2d-kube-api-access-r8rl6" pod "f403b27f-184b-42fd-839a-92fa3b4cdc2d" (UID: "f403b27f-184b-42fd-839a-92fa3b4cdc2d"). InnerVolumeSpecName "kube-api-access-r8rl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:27.978783 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.978763 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a819308-670a-4a83-8512-630337c61d5a-istio-token" pod "7a819308-670a-4a83-8512-630337c61d5a" (UID: "7a819308-670a-4a83-8512-630337c61d5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:27.978863 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.978840 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a819308-670a-4a83-8512-630337c61d5a-istio-podinfo" pod "7a819308-670a-4a83-8512-630337c61d5a" (UID: "7a819308-670a-4a83-8512-630337c61d5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:15:27.979249 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.979226 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-envoy" pod "f403b27f-184b-42fd-839a-92fa3b4cdc2d" (UID: "f403b27f-184b-42fd-839a-92fa3b4cdc2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:27.979373 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.979248 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-podinfo" pod "f403b27f-184b-42fd-839a-92fa3b4cdc2d" (UID: "f403b27f-184b-42fd-839a-92fa3b4cdc2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:15:27.979373 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.979252 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a819308-670a-4a83-8512-630337c61d5a-kube-api-access-vsl85" pod "7a819308-670a-4a83-8512-630337c61d5a" (UID: "7a819308-670a-4a83-8512-630337c61d5a"). InnerVolumeSpecName "kube-api-access-vsl85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:27.979911 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.979887 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-istio-envoy" pod "7a819308-670a-4a83-8512-630337c61d5a" (UID: "7a819308-670a-4a83-8512-630337c61d5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:27.980484 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:27.980457 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-token" pod "f403b27f-184b-42fd-839a-92fa3b4cdc2d" (UID: "f403b27f-184b-42fd-839a-92fa3b4cdc2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:28.076864 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.076542 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:28.076864 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.076568 5805 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-istio-envoy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:28.076864 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.076586 5805 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-envoy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:28.076864 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.076672 5805 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-podinfo\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:28.076864 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.076691 5805 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a819308-670a-4a83-8512-630337c61d5a-credential-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:28.076864 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.076706 5805 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istio-data\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:28.076864 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.076739 5805 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-workload-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:28.076864 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.076766 5805 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f403b27f-184b-42fd-839a-92fa3b4cdc2d-istiod-ca-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:28.076864 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.076780 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8rl6\" (UniqueName: \"kubernetes.io/projected/f403b27f-184b-42fd-839a-92fa3b4cdc2d-kube-api-access-r8rl6\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:28.076864 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.076796 5805 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f403b27f-184b-42fd-839a-92fa3b4cdc2d-workload-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:28.076864 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.076810 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a819308-670a-4a83-8512-630337c61d5a-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:28.076864 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.076839 5805 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a819308-670a-4a83-8512-630337c61d5a-istiod-ca-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:28.076864 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.076853 5805 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a819308-670a-4a83-8512-630337c61d5a-istio-podinfo\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:28.076864 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.076869 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsl85\" (UniqueName: \"kubernetes.io/projected/7a819308-670a-4a83-8512-630337c61d5a-kube-api-access-vsl85\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:28.681729 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.681697 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" Nov 03 18:15:28.681729 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.681716 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw" event={"ID":"7a819308-670a-4a83-8512-630337c61d5a","Type":"ContainerDied","Data":"fd84daf29a2e782b2a27368bbf627d96d99259e84e57a443a1730007e958c0ab"} Nov 03 18:15:28.682252 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.681811 5805 scope.go:122] "RemoveContainer" containerID="c6b9756f2536b09d7a58f7c3b85de440249c2f523e8e2bcab3a53fed2f27e604" Nov 03 18:15:28.688461 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.688431 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" event={"ID":"f403b27f-184b-42fd-839a-92fa3b4cdc2d","Type":"ContainerDied","Data":"97185dcbccd8b63a79ead35e6025ad50c10db052ed61f2be1f4cc1ff27f5c2b2"} Nov 03 18:15:28.688598 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.688447 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v" Nov 03 18:15:28.698325 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.698308 5805 scope.go:122] "RemoveContainer" containerID="a1ef37a20a86aeb6b14d713c064c54d9c5e4b4dd435cf111e33341c312ecce7c" Nov 03 18:15:28.836015 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.835988 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v"] Nov 03 18:15:28.881886 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.881860 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-xr52v"] Nov 03 18:15:28.941962 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.941897 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw"] Nov 03 18:15:28.950364 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.950331 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0adcbc-openshift-default-d9859d4d9-bkjnw"] Nov 03 18:15:28.979526 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.979491 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a819308-670a-4a83-8512-630337c61d5a" path="/var/lib/kubelet/pods/7a819308-670a-4a83-8512-630337c61d5a/volumes" Nov 03 18:15:28.980090 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.980068 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d997ac9-3961-424f-88c5-7b2c2168eab9" path="/var/lib/kubelet/pods/7d997ac9-3961-424f-88c5-7b2c2168eab9/volumes" Nov 03 18:15:28.980599 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.980571 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f403b27f-184b-42fd-839a-92fa3b4cdc2d" path="/var/lib/kubelet/pods/f403b27f-184b-42fd-839a-92fa3b4cdc2d/volumes" Nov 03 18:15:28.981922 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:28.981893 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:33.570188 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:33.570141 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:34.344504 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.344475 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n"] Nov 03 18:15:34.344695 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.344673 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" podUID="0fe4928a-7b15-472b-8c9e-fc28b4c46e43" containerName="discovery" containerID="cri-o://40d4b3d869ec85e486371787e979ad7624784a528175a64df25f149c9540d7dd" gracePeriod=30 Nov 03 18:15:34.615953 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.615927 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:15:34.729500 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.729461 5805 generic.go:356] "Generic (PLEG): container finished" podID="0fe4928a-7b15-472b-8c9e-fc28b4c46e43" containerID="40d4b3d869ec85e486371787e979ad7624784a528175a64df25f149c9540d7dd" exitCode=0 Nov 03 18:15:34.729688 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.729669 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" event={"ID":"0fe4928a-7b15-472b-8c9e-fc28b4c46e43","Type":"ContainerDied","Data":"40d4b3d869ec85e486371787e979ad7624784a528175a64df25f149c9540d7dd"} Nov 03 18:15:34.729757 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.729719 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" event={"ID":"0fe4928a-7b15-472b-8c9e-fc28b4c46e43","Type":"ContainerDied","Data":"b4d2591c6e3d09c2f5452cb882bb9991268c72698d1181ecb26901b5b335e058"} Nov 03 18:15:34.729757 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.729744 5805 scope.go:122] "RemoveContainer" containerID="40d4b3d869ec85e486371787e979ad7624784a528175a64df25f149c9540d7dd" Nov 03 18:15:34.729892 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.729878 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n" Nov 03 18:15:34.741359 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.741337 5805 scope.go:122] "RemoveContainer" containerID="40d4b3d869ec85e486371787e979ad7624784a528175a64df25f149c9540d7dd" Nov 03 18:15:34.741711 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:15:34.741686 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40d4b3d869ec85e486371787e979ad7624784a528175a64df25f149c9540d7dd\": container with ID starting with 40d4b3d869ec85e486371787e979ad7624784a528175a64df25f149c9540d7dd not found: ID does not exist" containerID="40d4b3d869ec85e486371787e979ad7624784a528175a64df25f149c9540d7dd" Nov 03 18:15:34.741820 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.741717 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d4b3d869ec85e486371787e979ad7624784a528175a64df25f149c9540d7dd"} err="failed to get container status \"40d4b3d869ec85e486371787e979ad7624784a528175a64df25f149c9540d7dd\": rpc error: code = NotFound desc = could not find container \"40d4b3d869ec85e486371787e979ad7624784a528175a64df25f149c9540d7dd\": container with ID starting with 40d4b3d869ec85e486371787e979ad7624784a528175a64df25f149c9540d7dd not found: ID does not exist" Nov 03 18:15:34.752129 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.752103 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-cacerts\" (UniqueName: \"kubernetes.io/secret/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-cacerts\") pod \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " Nov 03 18:15:34.752264 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.752172 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-local-certs\") pod \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " Nov 03 18:15:34.752264 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.752215 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-kubeconfig\") pod \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " Nov 03 18:15:34.752384 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.752314 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-kube-api-access-5sbf4\" (UniqueName: \"kubernetes.io/projected/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-kube-api-access-5sbf4\") pod \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " Nov 03 18:15:34.752384 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.752356 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-token\" (UniqueName: \"kubernetes.io/projected/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-token\") pod \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " Nov 03 18:15:34.752384 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.752379 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-csr-dns-cert\") pod \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " Nov 03 18:15:34.752579 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.752452 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-csr-ca-configmap\") pod \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\" (UID: \"0fe4928a-7b15-472b-8c9e-fc28b4c46e43\") " Nov 03 18:15:34.753277 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.753252 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-csr-ca-configmap" pod "0fe4928a-7b15-472b-8c9e-fc28b4c46e43" (UID: "0fe4928a-7b15-472b-8c9e-fc28b4c46e43"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:34.758719 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.758674 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-cacerts" pod "0fe4928a-7b15-472b-8c9e-fc28b4c46e43" (UID: "0fe4928a-7b15-472b-8c9e-fc28b4c46e43"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:15:34.758719 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.758674 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-token" pod "0fe4928a-7b15-472b-8c9e-fc28b4c46e43" (UID: "0fe4928a-7b15-472b-8c9e-fc28b4c46e43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:34.758719 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.758699 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-csr-dns-cert" pod "0fe4928a-7b15-472b-8c9e-fc28b4c46e43" (UID: "0fe4928a-7b15-472b-8c9e-fc28b4c46e43"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:15:34.758919 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.758717 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-kubeconfig" pod "0fe4928a-7b15-472b-8c9e-fc28b4c46e43" (UID: "0fe4928a-7b15-472b-8c9e-fc28b4c46e43"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:15:34.758919 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.758703 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-kube-api-access-5sbf4" pod "0fe4928a-7b15-472b-8c9e-fc28b4c46e43" (UID: "0fe4928a-7b15-472b-8c9e-fc28b4c46e43"). InnerVolumeSpecName "kube-api-access-5sbf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:34.759860 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.759836 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-local-certs" pod "0fe4928a-7b15-472b-8c9e-fc28b4c46e43" (UID: "0fe4928a-7b15-472b-8c9e-fc28b4c46e43"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:34.854658 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.854594 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:34.854658 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.854621 5805 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-csr-dns-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:34.854658 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.854630 5805 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-csr-ca-configmap\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:34.854658 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.854640 5805 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-cacerts\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:34.854658 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.854649 5805 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-local-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:34.854658 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.854657 5805 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-istio-kubeconfig\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:34.854924 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.854666 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sbf4\" (UniqueName: \"kubernetes.io/projected/0fe4928a-7b15-472b-8c9e-fc28b4c46e43-kube-api-access-5sbf4\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:34.893468 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:34.893344 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:35.137884 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:35.137803 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n"] Nov 03 18:15:35.183459 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:35.183427 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9mp9n"] Nov 03 18:15:36.092592 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.092561 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9"] Nov 03 18:15:36.093032 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093019 5805 cpu_manager.go:414] "containerMap: removing container" podUID="7d997ac9-3961-424f-88c5-7b2c2168eab9" containerName="istio-proxy" Nov 03 18:15:36.093077 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093032 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d997ac9-3961-424f-88c5-7b2c2168eab9" containerName="istio-proxy" Nov 03 18:15:36.093077 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093042 5805 cpu_manager.go:414] "containerMap: removing container" podUID="73900785-3360-4c42-8a5d-8c0e258fc637" containerName="cleanup" Nov 03 18:15:36.093077 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093048 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73900785-3360-4c42-8a5d-8c0e258fc637" containerName="cleanup" Nov 03 18:15:36.093077 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093056 5805 cpu_manager.go:414] "containerMap: removing container" podUID="b78a7ba2-1042-492c-b52c-dca73afd27d5" containerName="cleanup" Nov 03 18:15:36.093077 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093061 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b78a7ba2-1042-492c-b52c-dca73afd27d5" containerName="cleanup" Nov 03 18:15:36.093077 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093068 5805 cpu_manager.go:414] "containerMap: removing container" podUID="d22875d5-06bf-4abe-9961-9f5c9922eac0" containerName="cleanup" Nov 03 18:15:36.093077 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093073 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d22875d5-06bf-4abe-9961-9f5c9922eac0" containerName="cleanup" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093081 5805 cpu_manager.go:414] "containerMap: removing container" podUID="60483b95-19d2-46c3-ad70-5d8b36ffb572" containerName="cleanup" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093085 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60483b95-19d2-46c3-ad70-5d8b36ffb572" containerName="cleanup" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093095 5805 cpu_manager.go:414] "containerMap: removing container" podUID="7221deb2-4d2f-4f43-b7a5-064464fc3991" containerName="cleanup" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093099 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7221deb2-4d2f-4f43-b7a5-064464fc3991" containerName="cleanup" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093106 5805 cpu_manager.go:414] "containerMap: removing container" podUID="7221deb2-4d2f-4f43-b7a5-064464fc3991" containerName="cleanup" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093111 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7221deb2-4d2f-4f43-b7a5-064464fc3991" containerName="cleanup" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093118 5805 cpu_manager.go:414] "containerMap: removing container" podUID="7221deb2-4d2f-4f43-b7a5-064464fc3991" containerName="cleanup" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093123 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7221deb2-4d2f-4f43-b7a5-064464fc3991" containerName="cleanup" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093129 5805 cpu_manager.go:414] "containerMap: removing container" podUID="73900785-3360-4c42-8a5d-8c0e258fc637" containerName="cleanup" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093133 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73900785-3360-4c42-8a5d-8c0e258fc637" containerName="cleanup" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093139 5805 cpu_manager.go:414] "containerMap: removing container" podUID="277a1fec-5c82-449d-839f-e843fa0f813d" containerName="main" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093143 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277a1fec-5c82-449d-839f-e843fa0f813d" containerName="main" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093154 5805 cpu_manager.go:414] "containerMap: removing container" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerName="main" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093158 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerName="main" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093165 5805 cpu_manager.go:414] "containerMap: removing container" podUID="b78a7ba2-1042-492c-b52c-dca73afd27d5" containerName="cleanup" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093171 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b78a7ba2-1042-492c-b52c-dca73afd27d5" containerName="cleanup" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093182 5805 cpu_manager.go:414] "containerMap: removing container" podUID="7a819308-670a-4a83-8512-630337c61d5a" containerName="istio-proxy" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093186 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a819308-670a-4a83-8512-630337c61d5a" containerName="istio-proxy" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093193 5805 cpu_manager.go:414] "containerMap: removing container" podUID="51204cd4-e74e-4d85-acf7-2118f81797d2" containerName="istio-proxy" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093197 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51204cd4-e74e-4d85-acf7-2118f81797d2" containerName="istio-proxy" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093207 5805 cpu_manager.go:414] "containerMap: removing container" podUID="5efde3b1-24ad-4da6-81e9-09c7c7aa181c" containerName="istio-proxy" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093212 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5efde3b1-24ad-4da6-81e9-09c7c7aa181c" containerName="istio-proxy" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093220 5805 cpu_manager.go:414] "containerMap: removing container" podUID="d22875d5-06bf-4abe-9961-9f5c9922eac0" containerName="cleanup" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093224 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d22875d5-06bf-4abe-9961-9f5c9922eac0" containerName="cleanup" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093230 5805 cpu_manager.go:414] "containerMap: removing container" podUID="d22875d5-06bf-4abe-9961-9f5c9922eac0" containerName="cleanup" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093235 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d22875d5-06bf-4abe-9961-9f5c9922eac0" containerName="cleanup" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093243 5805 cpu_manager.go:414] "containerMap: removing container" podUID="f403b27f-184b-42fd-839a-92fa3b4cdc2d" containerName="istio-proxy" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093248 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f403b27f-184b-42fd-839a-92fa3b4cdc2d" containerName="istio-proxy" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093254 5805 cpu_manager.go:414] "containerMap: removing container" podUID="10efc538-5a7a-40ce-b6be-819a2cddbec3" containerName="istio-proxy" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093260 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10efc538-5a7a-40ce-b6be-819a2cddbec3" containerName="istio-proxy" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093272 5805 cpu_manager.go:414] "containerMap: removing container" podUID="00a224da-b74e-4186-b7ef-030cc5fc1c92" containerName="revoke-keys" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093276 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a224da-b74e-4186-b7ef-030cc5fc1c92" containerName="revoke-keys" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093284 5805 cpu_manager.go:414] "containerMap: removing container" podUID="ce5323ef-e7f2-4921-bed5-9e71e2aa835d" containerName="revoke-keys" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093289 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce5323ef-e7f2-4921-bed5-9e71e2aa835d" containerName="revoke-keys" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093296 5805 cpu_manager.go:414] "containerMap: removing container" podUID="0fe4928a-7b15-472b-8c9e-fc28b4c46e43" containerName="discovery" Nov 03 18:15:36.093292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093301 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe4928a-7b15-472b-8c9e-fc28b4c46e43" containerName="discovery" Nov 03 18:15:36.094460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093365 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d997ac9-3961-424f-88c5-7b2c2168eab9" containerName="istio-proxy" Nov 03 18:15:36.094460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093375 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="73900785-3360-4c42-8a5d-8c0e258fc637" containerName="cleanup" Nov 03 18:15:36.094460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093381 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="b78a7ba2-1042-492c-b52c-dca73afd27d5" containerName="cleanup" Nov 03 18:15:36.094460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093387 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="d22875d5-06bf-4abe-9961-9f5c9922eac0" containerName="cleanup" Nov 03 18:15:36.094460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093408 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="7221deb2-4d2f-4f43-b7a5-064464fc3991" containerName="cleanup" Nov 03 18:15:36.094460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093416 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="7221deb2-4d2f-4f43-b7a5-064464fc3991" containerName="cleanup" Nov 03 18:15:36.094460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093424 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="7221deb2-4d2f-4f43-b7a5-064464fc3991" containerName="cleanup" Nov 03 18:15:36.094460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093433 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="73900785-3360-4c42-8a5d-8c0e258fc637" containerName="cleanup" Nov 03 18:15:36.094460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093441 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="277a1fec-5c82-449d-839f-e843fa0f813d" containerName="main" Nov 03 18:15:36.094460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093451 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="82c4f5f3-b386-4ade-bfc9-d5de175e5bcc" containerName="main" Nov 03 18:15:36.094460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093458 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="b78a7ba2-1042-492c-b52c-dca73afd27d5" containerName="cleanup" Nov 03 18:15:36.094460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093466 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a819308-670a-4a83-8512-630337c61d5a" containerName="istio-proxy" Nov 03 18:15:36.094460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093472 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="51204cd4-e74e-4d85-acf7-2118f81797d2" containerName="istio-proxy" Nov 03 18:15:36.094460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093478 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="73900785-3360-4c42-8a5d-8c0e258fc637" containerName="cleanup" Nov 03 18:15:36.094460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093502 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="5efde3b1-24ad-4da6-81e9-09c7c7aa181c" containerName="istio-proxy" Nov 03 18:15:36.094460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093509 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="d22875d5-06bf-4abe-9961-9f5c9922eac0" containerName="cleanup" Nov 03 18:15:36.094460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093515 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="f403b27f-184b-42fd-839a-92fa3b4cdc2d" containerName="istio-proxy" Nov 03 18:15:36.094460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093522 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="10efc538-5a7a-40ce-b6be-819a2cddbec3" containerName="istio-proxy" Nov 03 18:15:36.094460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093527 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="b78a7ba2-1042-492c-b52c-dca73afd27d5" containerName="cleanup" Nov 03 18:15:36.094460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093535 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a224da-b74e-4186-b7ef-030cc5fc1c92" containerName="revoke-keys" Nov 03 18:15:36.094460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093541 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce5323ef-e7f2-4921-bed5-9e71e2aa835d" containerName="revoke-keys" Nov 03 18:15:36.094460 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.093547 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe4928a-7b15-472b-8c9e-fc28b4c46e43" containerName="discovery" Nov 03 18:15:36.098086 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.098065 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9" Nov 03 18:15:36.148624 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.148603 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:15:36.169492 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.169460 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8664b8f5-f2eb-4d49-84d6-d568be08e781-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9\" (UID: \"8664b8f5-f2eb-4d49-84d6-d568be08e781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9" Nov 03 18:15:36.169658 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.169635 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc5nm\" (UniqueName: \"kubernetes.io/projected/8664b8f5-f2eb-4d49-84d6-d568be08e781-kube-api-access-tc5nm\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9\" (UID: \"8664b8f5-f2eb-4d49-84d6-d568be08e781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9" Nov 03 18:15:36.169723 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.169677 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8664b8f5-f2eb-4d49-84d6-d568be08e781-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9\" (UID: \"8664b8f5-f2eb-4d49-84d6-d568be08e781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9" Nov 03 18:15:36.194825 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.194800 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\"" Nov 03 18:15:36.197222 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.197201 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9"] Nov 03 18:15:36.271254 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.271221 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc5nm\" (UniqueName: \"kubernetes.io/projected/8664b8f5-f2eb-4d49-84d6-d568be08e781-kube-api-access-tc5nm\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9\" (UID: \"8664b8f5-f2eb-4d49-84d6-d568be08e781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9" Nov 03 18:15:36.271391 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.271259 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8664b8f5-f2eb-4d49-84d6-d568be08e781-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9\" (UID: \"8664b8f5-f2eb-4d49-84d6-d568be08e781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9" Nov 03 18:15:36.271391 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.271366 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8664b8f5-f2eb-4d49-84d6-d568be08e781-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9\" (UID: \"8664b8f5-f2eb-4d49-84d6-d568be08e781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9" Nov 03 18:15:36.272037 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.272015 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8664b8f5-f2eb-4d49-84d6-d568be08e781-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9\" (UID: \"8664b8f5-f2eb-4d49-84d6-d568be08e781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9" Nov 03 18:15:36.273613 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.273594 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8664b8f5-f2eb-4d49-84d6-d568be08e781-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9\" (UID: \"8664b8f5-f2eb-4d49-84d6-d568be08e781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9" Nov 03 18:15:36.329922 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.329895 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc5nm\" (UniqueName: \"kubernetes.io/projected/8664b8f5-f2eb-4d49-84d6-d568be08e781-kube-api-access-tc5nm\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9\" (UID: \"8664b8f5-f2eb-4d49-84d6-d568be08e781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9" Nov 03 18:15:36.411434 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.411378 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9" Nov 03 18:15:36.784972 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.784899 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9"] Nov 03 18:15:36.790913 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:15:36.790882 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8664b8f5_f2eb_4d49_84d6_d568be08e781.slice/crio-cad6a715df454777c1cc86036168658bfab062bfdfc2c380c5c17ed735f65d63 WatchSource:0}: Error finding container cad6a715df454777c1cc86036168658bfab062bfdfc2c380c5c17ed735f65d63: Status 404 returned error can't find the container with id cad6a715df454777c1cc86036168658bfab062bfdfc2c380c5c17ed735f65d63 Nov 03 18:15:36.977768 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:36.977739 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fe4928a-7b15-472b-8c9e-fc28b4c46e43" path="/var/lib/kubelet/pods/0fe4928a-7b15-472b-8c9e-fc28b4c46e43/volumes" Nov 03 18:15:37.752957 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:37.752927 5805 generic.go:356] "Generic (PLEG): container finished" podID="8664b8f5-f2eb-4d49-84d6-d568be08e781" containerID="85e0a826c32c9d2125af7225ba0692609093410d5bad6ac7076952427655c743" exitCode=0 Nov 03 18:15:37.753327 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:37.753000 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9" event={"ID":"8664b8f5-f2eb-4d49-84d6-d568be08e781","Type":"ContainerDied","Data":"85e0a826c32c9d2125af7225ba0692609093410d5bad6ac7076952427655c743"} Nov 03 18:15:37.753327 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:37.753086 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9" event={"ID":"8664b8f5-f2eb-4d49-84d6-d568be08e781","Type":"ContainerStarted","Data":"cad6a715df454777c1cc86036168658bfab062bfdfc2c380c5c17ed735f65d63"} Nov 03 18:15:38.304331 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:38.304293 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:38.907484 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:38.907442 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9" Nov 03 18:15:39.001869 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:39.001834 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8664b8f5-f2eb-4d49-84d6-d568be08e781-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8664b8f5-f2eb-4d49-84d6-d568be08e781-maas-api-openshift-ca\") pod \"8664b8f5-f2eb-4d49-84d6-d568be08e781\" (UID: \"8664b8f5-f2eb-4d49-84d6-d568be08e781\") " Nov 03 18:15:39.002031 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:39.001898 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8664b8f5-f2eb-4d49-84d6-d568be08e781-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8664b8f5-f2eb-4d49-84d6-d568be08e781-maas-api-xks-ca\") pod \"8664b8f5-f2eb-4d49-84d6-d568be08e781\" (UID: \"8664b8f5-f2eb-4d49-84d6-d568be08e781\") " Nov 03 18:15:39.002031 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:39.001942 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8664b8f5-f2eb-4d49-84d6-d568be08e781-kube-api-access-tc5nm\" (UniqueName: \"kubernetes.io/projected/8664b8f5-f2eb-4d49-84d6-d568be08e781-kube-api-access-tc5nm\") pod \"8664b8f5-f2eb-4d49-84d6-d568be08e781\" (UID: \"8664b8f5-f2eb-4d49-84d6-d568be08e781\") " Nov 03 18:15:39.002212 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:39.002187 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8664b8f5-f2eb-4d49-84d6-d568be08e781-maas-api-openshift-ca" pod "8664b8f5-f2eb-4d49-84d6-d568be08e781" (UID: "8664b8f5-f2eb-4d49-84d6-d568be08e781"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:39.002583 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:39.002562 5805 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8664b8f5-f2eb-4d49-84d6-d568be08e781-maas-api-openshift-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:39.004145 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:39.004121 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8664b8f5-f2eb-4d49-84d6-d568be08e781-maas-api-xks-ca" pod "8664b8f5-f2eb-4d49-84d6-d568be08e781" (UID: "8664b8f5-f2eb-4d49-84d6-d568be08e781"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:15:39.004248 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:39.004233 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8664b8f5-f2eb-4d49-84d6-d568be08e781-kube-api-access-tc5nm" pod "8664b8f5-f2eb-4d49-84d6-d568be08e781" (UID: "8664b8f5-f2eb-4d49-84d6-d568be08e781"). InnerVolumeSpecName "kube-api-access-tc5nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:39.103520 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:39.103483 5805 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8664b8f5-f2eb-4d49-84d6-d568be08e781-maas-api-xks-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:39.103520 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:39.103516 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc5nm\" (UniqueName: \"kubernetes.io/projected/8664b8f5-f2eb-4d49-84d6-d568be08e781-kube-api-access-tc5nm\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:39.768134 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:39.768101 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9" Nov 03 18:15:39.768134 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:39.768126 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9" event={"ID":"8664b8f5-f2eb-4d49-84d6-d568be08e781","Type":"ContainerDied","Data":"cad6a715df454777c1cc86036168658bfab062bfdfc2c380c5c17ed735f65d63"} Nov 03 18:15:39.768413 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:39.768173 5805 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cad6a715df454777c1cc86036168658bfab062bfdfc2c380c5c17ed735f65d63" Nov 03 18:15:41.503445 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:41.503367 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:41.706100 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:41.706070 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:42.975224 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:42.975188 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:47.458242 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:47.458194 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:47.651232 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:47.651195 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:47.709739 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:47.709647 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h"] Nov 03 18:15:48.025607 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:48.025500 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:48.236249 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:48.236218 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:51.363583 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:51.363545 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:52.453470 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:52.453433 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:52.710882 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:52.710792 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h" podUID="36b2f655-b26e-4a96-a114-5596317d889e" containerName="payload-pre-processing" containerID="cri-o://5e31b7160f8a711f78caf347ce5e5c76e4de0bfa5e438d32d278f06c299e801d" gracePeriod=25 Nov 03 18:15:52.859424 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:52.859378 5805 generic.go:356] "Generic (PLEG): container finished" podID="36b2f655-b26e-4a96-a114-5596317d889e" containerID="5e31b7160f8a711f78caf347ce5e5c76e4de0bfa5e438d32d278f06c299e801d" exitCode=0 Nov 03 18:15:52.859607 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:52.859422 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h" event={"ID":"36b2f655-b26e-4a96-a114-5596317d889e","Type":"ContainerDied","Data":"5e31b7160f8a711f78caf347ce5e5c76e4de0bfa5e438d32d278f06c299e801d"} Nov 03 18:15:52.873169 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:52.873120 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:52.983207 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:52.983180 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h" Nov 03 18:15:53.000851 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:53.000820 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b2f655-b26e-4a96-a114-5596317d889e-kube-api-access-hq795\" (UniqueName: \"kubernetes.io/projected/36b2f655-b26e-4a96-a114-5596317d889e-kube-api-access-hq795\") pod \"36b2f655-b26e-4a96-a114-5596317d889e\" (UID: \"36b2f655-b26e-4a96-a114-5596317d889e\") " Nov 03 18:15:53.001008 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:53.000872 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b2f655-b26e-4a96-a114-5596317d889e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b2f655-b26e-4a96-a114-5596317d889e-plugins-config-volume\") pod \"36b2f655-b26e-4a96-a114-5596317d889e\" (UID: \"36b2f655-b26e-4a96-a114-5596317d889e\") " Nov 03 18:15:53.001275 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:53.001250 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b2f655-b26e-4a96-a114-5596317d889e-plugins-config-volume" pod "36b2f655-b26e-4a96-a114-5596317d889e" (UID: "36b2f655-b26e-4a96-a114-5596317d889e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:53.001497 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:53.001479 5805 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36b2f655-b26e-4a96-a114-5596317d889e-plugins-config-volume\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:53.002837 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:53.002819 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b2f655-b26e-4a96-a114-5596317d889e-kube-api-access-hq795" pod "36b2f655-b26e-4a96-a114-5596317d889e" (UID: "36b2f655-b26e-4a96-a114-5596317d889e"). InnerVolumeSpecName "kube-api-access-hq795". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:53.102681 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:53.102646 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq795\" (UniqueName: \"kubernetes.io/projected/36b2f655-b26e-4a96-a114-5596317d889e-kube-api-access-hq795\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:15:53.103000 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:53.102978 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:53.555335 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:53.555300 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:53.763906 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:53.763875 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:53.873060 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:53.873021 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h" event={"ID":"36b2f655-b26e-4a96-a114-5596317d889e","Type":"ContainerDied","Data":"e4cac485a5c24543c35193b13527e0dd433a090e16ff61169bcd02448aab9f13"} Nov 03 18:15:53.873245 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:53.873100 5805 scope.go:122] "RemoveContainer" containerID="5e31b7160f8a711f78caf347ce5e5c76e4de0bfa5e438d32d278f06c299e801d" Nov 03 18:15:53.873302 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:53.873264 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h" Nov 03 18:15:53.972408 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:53.972333 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:54.050049 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:54.050015 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h"] Nov 03 18:15:54.095146 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:54.095119 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7a7c76-7b64fdc9bc-ddq9h"] Nov 03 18:15:54.171384 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:54.171305 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:54.372067 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:54.372036 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:54.762270 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:54.762239 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:54.975638 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:54.975608 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36b2f655-b26e-4a96-a114-5596317d889e" path="/var/lib/kubelet/pods/36b2f655-b26e-4a96-a114-5596317d889e/volumes" Nov 03 18:15:55.034133 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:55.034058 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:55.277734 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:55.277701 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:55.473665 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:55.473626 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:55.525810 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:55.525768 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9"] Nov 03 18:15:55.571583 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:55.571554 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7a7c76-fr2f9"] Nov 03 18:15:56.040248 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:56.040217 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:15:56.235079 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:15:56.235038 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:15:56.976859 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:15:56.976824 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8664b8f5-f2eb-4d49-84d6-d568be08e781" path="/var/lib/kubelet/pods/8664b8f5-f2eb-4d49-84d6-d568be08e781/volumes" Nov 03 18:16:00.194439 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:00.194402 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:00.835600 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:00.835561 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:01.061862 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:01.061745 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:01.303956 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:01.303840 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:01.880192 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:01.879986 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:02.091555 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:02.091520 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:02.174659 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:02.174578 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t"] Nov 03 18:16:02.174814 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:02.174797 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" podUID="e4120351-f702-437b-aa9b-40979fa30455" containerName="istio-proxy" containerID="cri-o://2cc75fa9b42ceca8837288cdba652af7aadaf3c2919b039b404590272a8bf1f7" gracePeriod=30 Nov 03 18:16:02.188875 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:02.188847 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk"] Nov 03 18:16:02.189110 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:02.189083 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" podUID="94ef80bb-de9b-4163-9db7-0fd6f19a8eca" containerName="istio-proxy" containerID="cri-o://5ce052d145555942ce3cd15c2961039306daa4df02f4f26c51e2b2f4d2111f55" gracePeriod=30 Nov 03 18:16:02.314377 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:02.314330 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:02.689827 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:02.689791 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:02.901583 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:02.901550 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:03.112417 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:03.112352 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:03.265032 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:03.265003 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 18:16:03.267698 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:03.267678 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 18:16:05.200996 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:05.200920 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:05.673756 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:05.673723 5805 patch_prober.go:29] interesting pod/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:16:05.673914 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:05.673776 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" podUID="e4120351-f702-437b-aa9b-40979fa30455" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:16:06.353930 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:06.353886 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:06.548675 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:06.548623 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:07.461134 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.461103 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:16:07.497304 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.497281 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:16:07.579511 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.579476 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-credential-socket\") pod \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " Nov 03 18:16:07.579716 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.579540 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4120351-f702-437b-aa9b-40979fa30455-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4120351-f702-437b-aa9b-40979fa30455-istio-podinfo\") pod \"e4120351-f702-437b-aa9b-40979fa30455\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " Nov 03 18:16:07.579716 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.579571 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4120351-f702-437b-aa9b-40979fa30455-kube-api-access-cbsll\" (UniqueName: \"kubernetes.io/projected/e4120351-f702-437b-aa9b-40979fa30455-kube-api-access-cbsll\") pod \"e4120351-f702-437b-aa9b-40979fa30455\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " Nov 03 18:16:07.579716 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.579706 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4120351-f702-437b-aa9b-40979fa30455-istio-token\" (UniqueName: \"kubernetes.io/projected/e4120351-f702-437b-aa9b-40979fa30455-istio-token\") pod \"e4120351-f702-437b-aa9b-40979fa30455\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " Nov 03 18:16:07.579923 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.579742 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-workload-certs\") pod \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " Nov 03 18:16:07.579923 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.579778 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-workload-socket\") pod \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " Nov 03 18:16:07.579923 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.579813 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-istio-data\") pod \"e4120351-f702-437b-aa9b-40979fa30455\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " Nov 03 18:16:07.579923 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.579864 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istiod-ca-cert\") pod \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " Nov 03 18:16:07.579923 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.579921 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-credential-socket\") pod \"e4120351-f702-437b-aa9b-40979fa30455\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " Nov 03 18:16:07.580223 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.579964 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-envoy\") pod \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " Nov 03 18:16:07.580223 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.580014 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-workload-certs\") pod \"e4120351-f702-437b-aa9b-40979fa30455\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " Nov 03 18:16:07.580223 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.580097 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-credential-socket" pod "94ef80bb-de9b-4163-9db7-0fd6f19a8eca" (UID: "94ef80bb-de9b-4163-9db7-0fd6f19a8eca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:16:07.580420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.580253 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-workload-socket" pod "94ef80bb-de9b-4163-9db7-0fd6f19a8eca" (UID: "94ef80bb-de9b-4163-9db7-0fd6f19a8eca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:16:07.580420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.580409 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-credential-socket" pod "e4120351-f702-437b-aa9b-40979fa30455" (UID: "e4120351-f702-437b-aa9b-40979fa30455"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:16:07.580549 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.580470 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-istio-data" pod "e4120351-f702-437b-aa9b-40979fa30455" (UID: "e4120351-f702-437b-aa9b-40979fa30455"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:16:07.580549 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.580507 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-workload-certs" pod "94ef80bb-de9b-4163-9db7-0fd6f19a8eca" (UID: "94ef80bb-de9b-4163-9db7-0fd6f19a8eca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:16:07.580839 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.580702 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-workload-certs" pod "e4120351-f702-437b-aa9b-40979fa30455" (UID: "e4120351-f702-437b-aa9b-40979fa30455"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:16:07.580839 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.580813 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istiod-ca-cert" pod "94ef80bb-de9b-4163-9db7-0fd6f19a8eca" (UID: "94ef80bb-de9b-4163-9db7-0fd6f19a8eca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:16:07.581078 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.580883 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-istio-envoy\") pod \"e4120351-f702-437b-aa9b-40979fa30455\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " Nov 03 18:16:07.581078 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.580925 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-kube-api-access-4nvqj\" (UniqueName: \"kubernetes.io/projected/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-kube-api-access-4nvqj\") pod \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " Nov 03 18:16:07.581078 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.580957 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-token\" (UniqueName: \"kubernetes.io/projected/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-token\") pod \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " Nov 03 18:16:07.581078 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.580988 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-podinfo\") pod \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " Nov 03 18:16:07.581078 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.581015 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4120351-f702-437b-aa9b-40979fa30455-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4120351-f702-437b-aa9b-40979fa30455-istiod-ca-cert\") pod \"e4120351-f702-437b-aa9b-40979fa30455\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " Nov 03 18:16:07.581078 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.581072 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-workload-socket\") pod \"e4120351-f702-437b-aa9b-40979fa30455\" (UID: \"e4120351-f702-437b-aa9b-40979fa30455\") " Nov 03 18:16:07.581448 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.581168 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-data\") pod \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\" (UID: \"94ef80bb-de9b-4163-9db7-0fd6f19a8eca\") " Nov 03 18:16:07.581684 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.581661 5805 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-workload-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:16:07.581684 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.581683 5805 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-workload-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:16:07.581819 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.581697 5805 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-istio-data\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:16:07.581819 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.581710 5805 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istiod-ca-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:16:07.581819 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.581723 5805 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-credential-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:16:07.581819 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.581735 5805 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-workload-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:16:07.581819 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.581749 5805 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-credential-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:16:07.582080 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.581952 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-data" pod "94ef80bb-de9b-4163-9db7-0fd6f19a8eca" (UID: "94ef80bb-de9b-4163-9db7-0fd6f19a8eca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:16:07.582224 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.582203 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4120351-f702-437b-aa9b-40979fa30455-istiod-ca-cert" pod "e4120351-f702-437b-aa9b-40979fa30455" (UID: "e4120351-f702-437b-aa9b-40979fa30455"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:16:07.582487 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.582463 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-workload-socket" pod "e4120351-f702-437b-aa9b-40979fa30455" (UID: "e4120351-f702-437b-aa9b-40979fa30455"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:16:07.585069 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.585030 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-podinfo" pod "94ef80bb-de9b-4163-9db7-0fd6f19a8eca" (UID: "94ef80bb-de9b-4163-9db7-0fd6f19a8eca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:16:07.585189 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.585101 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-envoy" pod "94ef80bb-de9b-4163-9db7-0fd6f19a8eca" (UID: "94ef80bb-de9b-4163-9db7-0fd6f19a8eca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:16:07.585189 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.585157 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-kube-api-access-4nvqj" pod "94ef80bb-de9b-4163-9db7-0fd6f19a8eca" (UID: "94ef80bb-de9b-4163-9db7-0fd6f19a8eca"). InnerVolumeSpecName "kube-api-access-4nvqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:16:07.585316 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.585259 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-istio-envoy" pod "e4120351-f702-437b-aa9b-40979fa30455" (UID: "e4120351-f702-437b-aa9b-40979fa30455"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:16:07.585972 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.585951 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-token" pod "94ef80bb-de9b-4163-9db7-0fd6f19a8eca" (UID: "94ef80bb-de9b-4163-9db7-0fd6f19a8eca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:16:07.586710 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.586682 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4120351-f702-437b-aa9b-40979fa30455-istio-podinfo" pod "e4120351-f702-437b-aa9b-40979fa30455" (UID: "e4120351-f702-437b-aa9b-40979fa30455"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:16:07.586710 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.586706 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4120351-f702-437b-aa9b-40979fa30455-kube-api-access-cbsll" pod "e4120351-f702-437b-aa9b-40979fa30455" (UID: "e4120351-f702-437b-aa9b-40979fa30455"). InnerVolumeSpecName "kube-api-access-cbsll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:16:07.587808 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.587786 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4120351-f702-437b-aa9b-40979fa30455-istio-token" pod "e4120351-f702-437b-aa9b-40979fa30455" (UID: "e4120351-f702-437b-aa9b-40979fa30455"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:16:07.683105 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.683071 5805 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-istio-envoy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:16:07.683105 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.683103 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nvqj\" (UniqueName: \"kubernetes.io/projected/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-kube-api-access-4nvqj\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:16:07.683105 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.683113 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:16:07.683334 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.683123 5805 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-podinfo\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:16:07.683334 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.683134 5805 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4120351-f702-437b-aa9b-40979fa30455-istiod-ca-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:16:07.683334 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.683142 5805 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4120351-f702-437b-aa9b-40979fa30455-workload-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:16:07.683334 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.683153 5805 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-data\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:16:07.683334 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.683166 5805 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4120351-f702-437b-aa9b-40979fa30455-istio-podinfo\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:16:07.683334 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.683175 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbsll\" (UniqueName: \"kubernetes.io/projected/e4120351-f702-437b-aa9b-40979fa30455-kube-api-access-cbsll\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:16:07.683334 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.683184 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4120351-f702-437b-aa9b-40979fa30455-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:16:07.683334 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.683193 5805 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ef80bb-de9b-4163-9db7-0fd6f19a8eca-istio-envoy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:16:07.992951 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.992909 5805 generic.go:356] "Generic (PLEG): container finished" podID="e4120351-f702-437b-aa9b-40979fa30455" containerID="2cc75fa9b42ceca8837288cdba652af7aadaf3c2919b039b404590272a8bf1f7" exitCode=0 Nov 03 18:16:07.993148 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.993008 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" Nov 03 18:16:07.993148 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.993028 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" event={"ID":"e4120351-f702-437b-aa9b-40979fa30455","Type":"ContainerDied","Data":"2cc75fa9b42ceca8837288cdba652af7aadaf3c2919b039b404590272a8bf1f7"} Nov 03 18:16:07.993148 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.993088 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t" event={"ID":"e4120351-f702-437b-aa9b-40979fa30455","Type":"ContainerDied","Data":"b8a6a95fed779b28d2a1a8d34c4708a26a8fd058a6e75b4611d41afcddf56452"} Nov 03 18:16:07.993148 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.993123 5805 scope.go:122] "RemoveContainer" containerID="2cc75fa9b42ceca8837288cdba652af7aadaf3c2919b039b404590272a8bf1f7" Nov 03 18:16:07.994741 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.994713 5805 generic.go:356] "Generic (PLEG): container finished" podID="94ef80bb-de9b-4163-9db7-0fd6f19a8eca" containerID="5ce052d145555942ce3cd15c2961039306daa4df02f4f26c51e2b2f4d2111f55" exitCode=0 Nov 03 18:16:07.994875 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.994770 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" Nov 03 18:16:07.994875 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.994817 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" event={"ID":"94ef80bb-de9b-4163-9db7-0fd6f19a8eca","Type":"ContainerDied","Data":"5ce052d145555942ce3cd15c2961039306daa4df02f4f26c51e2b2f4d2111f55"} Nov 03 18:16:07.994973 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:07.994878 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk" event={"ID":"94ef80bb-de9b-4163-9db7-0fd6f19a8eca","Type":"ContainerDied","Data":"cd7977001742f84dc86b4de84b5581b779cd0a55fe780e59eb5aacd9b0a61f2f"} Nov 03 18:16:08.004709 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:08.004683 5805 scope.go:122] "RemoveContainer" containerID="2cc75fa9b42ceca8837288cdba652af7aadaf3c2919b039b404590272a8bf1f7" Nov 03 18:16:08.004977 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:16:08.004955 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cc75fa9b42ceca8837288cdba652af7aadaf3c2919b039b404590272a8bf1f7\": container with ID starting with 2cc75fa9b42ceca8837288cdba652af7aadaf3c2919b039b404590272a8bf1f7 not found: ID does not exist" containerID="2cc75fa9b42ceca8837288cdba652af7aadaf3c2919b039b404590272a8bf1f7" Nov 03 18:16:08.005044 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:08.004986 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cc75fa9b42ceca8837288cdba652af7aadaf3c2919b039b404590272a8bf1f7"} err="failed to get container status \"2cc75fa9b42ceca8837288cdba652af7aadaf3c2919b039b404590272a8bf1f7\": rpc error: code = NotFound desc = could not find container \"2cc75fa9b42ceca8837288cdba652af7aadaf3c2919b039b404590272a8bf1f7\": container with ID starting with 2cc75fa9b42ceca8837288cdba652af7aadaf3c2919b039b404590272a8bf1f7 not found: ID does not exist" Nov 03 18:16:08.005044 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:08.005013 5805 scope.go:122] "RemoveContainer" containerID="5ce052d145555942ce3cd15c2961039306daa4df02f4f26c51e2b2f4d2111f55" Nov 03 18:16:08.014956 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:08.014938 5805 scope.go:122] "RemoveContainer" containerID="5ce052d145555942ce3cd15c2961039306daa4df02f4f26c51e2b2f4d2111f55" Nov 03 18:16:08.015202 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:16:08.015182 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ce052d145555942ce3cd15c2961039306daa4df02f4f26c51e2b2f4d2111f55\": container with ID starting with 5ce052d145555942ce3cd15c2961039306daa4df02f4f26c51e2b2f4d2111f55 not found: ID does not exist" containerID="5ce052d145555942ce3cd15c2961039306daa4df02f4f26c51e2b2f4d2111f55" Nov 03 18:16:08.015279 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:08.015208 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ce052d145555942ce3cd15c2961039306daa4df02f4f26c51e2b2f4d2111f55"} err="failed to get container status \"5ce052d145555942ce3cd15c2961039306daa4df02f4f26c51e2b2f4d2111f55\": rpc error: code = NotFound desc = could not find container \"5ce052d145555942ce3cd15c2961039306daa4df02f4f26c51e2b2f4d2111f55\": container with ID starting with 5ce052d145555942ce3cd15c2961039306daa4df02f4f26c51e2b2f4d2111f55 not found: ID does not exist" Nov 03 18:16:08.151105 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:08.151068 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t"] Nov 03 18:16:08.198468 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:08.198431 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-s2c7t"] Nov 03 18:16:08.257716 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:08.257629 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk"] Nov 03 18:16:08.261973 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:08.261945 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a86228-openshift-default-58c566f45f-4cmvk"] Nov 03 18:16:08.522769 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:08.521277 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:08.730709 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:08.730672 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:08.976899 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:08.976859 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94ef80bb-de9b-4163-9db7-0fd6f19a8eca" path="/var/lib/kubelet/pods/94ef80bb-de9b-4163-9db7-0fd6f19a8eca/volumes" Nov 03 18:16:08.977445 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:08.977424 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4120351-f702-437b-aa9b-40979fa30455" path="/var/lib/kubelet/pods/e4120351-f702-437b-aa9b-40979fa30455/volumes" Nov 03 18:16:09.444134 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:09.444097 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:10.496888 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:10.496858 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:10.846919 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:10.846885 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:11.118384 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:11.118277 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:12.245936 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:12.245853 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:15.491727 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:15.491693 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:15.691269 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:15.691235 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:15.905517 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:15.905481 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:16.962072 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:16.962037 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:19.178837 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:19.178798 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:19.393861 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:19.393825 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:21.511868 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:21.511837 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:21.710450 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:21.710421 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:21.951591 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:21.951553 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:26.235308 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:16:26.235273 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:16:26.559950 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:26.559920 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:31.068223 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:31.068186 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:35.007328 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:35.007258 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:36.402698 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:36.402666 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:36.603759 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:36.603724 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:36.801654 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:36.801570 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:37.299268 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:37.299238 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:42.151459 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:42.151419 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:42.361943 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:42.361909 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:42.566796 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:42.566767 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:43.051633 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:43.051547 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:43.555943 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:43.555896 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:43.754434 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:43.754379 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:43.961101 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:43.961067 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:44.402858 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:44.402826 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:47.105792 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:47.105752 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:49.337783 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:49.337750 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz"] Nov 03 18:16:49.533827 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:49.533791 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:49.734871 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:49.734785 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:49.932331 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:49.932280 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:52.304280 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:52.304207 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:56.235872 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:16:56.235834 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:16:56.778416 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:56.774082 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:56.983007 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:56.982970 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:16:57.177079 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:16:57.177046 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:01.470589 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:01.470552 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:02.585744 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:02.585707 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:02.868233 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:02.868152 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:04.357968 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:04.357897 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" podUID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" containerName="main" containerID="cri-o://93bb816e462ef068a6af9188705606d14ac355598cfa13d717696e17d82e3bfd" gracePeriod=45 Nov 03 18:17:04.623292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:04.623217 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 18:17:04.742719 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:04.742683 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-model-cache\") pod \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\" (UID: \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\") " Nov 03 18:17:04.742912 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:04.742747 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-tmp-dir\") pod \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\" (UID: \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\") " Nov 03 18:17:04.742912 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:04.742838 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad8cc84a-716a-44c7-a257-1f630c48e1b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8cc84a-716a-44c7-a257-1f630c48e1b6-tls-certs\") pod \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\" (UID: \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\") " Nov 03 18:17:04.742912 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:04.742873 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-home\" (UniqueName: \"kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-home\") pod \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\" (UID: \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\") " Nov 03 18:17:04.743082 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:04.742943 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-dshm\") pod \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\" (UID: \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\") " Nov 03 18:17:04.743082 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:04.743001 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-model-cache" pod "ad8cc84a-716a-44c7-a257-1f630c48e1b6" (UID: "ad8cc84a-716a-44c7-a257-1f630c48e1b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:17:04.743082 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:04.743050 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad8cc84a-716a-44c7-a257-1f630c48e1b6-kube-api-access-6glml\" (UniqueName: \"kubernetes.io/projected/ad8cc84a-716a-44c7-a257-1f630c48e1b6-kube-api-access-6glml\") pod \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\" (UID: \"ad8cc84a-716a-44c7-a257-1f630c48e1b6\") " Nov 03 18:17:04.743248 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:04.743097 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-tmp-dir" pod "ad8cc84a-716a-44c7-a257-1f630c48e1b6" (UID: "ad8cc84a-716a-44c7-a257-1f630c48e1b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:17:04.743306 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:04.743240 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-home" pod "ad8cc84a-716a-44c7-a257-1f630c48e1b6" (UID: "ad8cc84a-716a-44c7-a257-1f630c48e1b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:17:04.743539 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:04.743512 5805 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-tmp-dir\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:04.743539 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:04.743538 5805 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-home\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:04.743724 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:04.743550 5805 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-model-cache\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:04.745275 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:04.745246 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-dshm" pod "ad8cc84a-716a-44c7-a257-1f630c48e1b6" (UID: "ad8cc84a-716a-44c7-a257-1f630c48e1b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:17:04.745432 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:04.745412 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad8cc84a-716a-44c7-a257-1f630c48e1b6-tls-certs" pod "ad8cc84a-716a-44c7-a257-1f630c48e1b6" (UID: "ad8cc84a-716a-44c7-a257-1f630c48e1b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:17:04.745689 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:04.745673 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad8cc84a-716a-44c7-a257-1f630c48e1b6-kube-api-access-6glml" pod "ad8cc84a-716a-44c7-a257-1f630c48e1b6" (UID: "ad8cc84a-716a-44c7-a257-1f630c48e1b6"). InnerVolumeSpecName "kube-api-access-6glml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:17:04.844633 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:04.844599 5805 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad8cc84a-716a-44c7-a257-1f630c48e1b6-dshm\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:04.844633 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:04.844631 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6glml\" (UniqueName: \"kubernetes.io/projected/ad8cc84a-716a-44c7-a257-1f630c48e1b6-kube-api-access-6glml\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:04.844826 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:04.844641 5805 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8cc84a-716a-44c7-a257-1f630c48e1b6-tls-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:05.339160 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:05.339114 5805 generic.go:356] "Generic (PLEG): container finished" podID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" containerID="93bb816e462ef068a6af9188705606d14ac355598cfa13d717696e17d82e3bfd" exitCode=0 Nov 03 18:17:05.339362 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:05.339172 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" Nov 03 18:17:05.339362 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:05.339205 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" event={"ID":"ad8cc84a-716a-44c7-a257-1f630c48e1b6","Type":"ContainerDied","Data":"93bb816e462ef068a6af9188705606d14ac355598cfa13d717696e17d82e3bfd"} Nov 03 18:17:05.339362 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:05.339256 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz" event={"ID":"ad8cc84a-716a-44c7-a257-1f630c48e1b6","Type":"ContainerDied","Data":"6b6613a89a7c67c24b8b6e45af1eb43c1a1ed4619d5860e73159f885d1c5a5bb"} Nov 03 18:17:05.339362 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:05.339283 5805 scope.go:122] "RemoveContainer" containerID="93bb816e462ef068a6af9188705606d14ac355598cfa13d717696e17d82e3bfd" Nov 03 18:17:05.351054 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:05.351033 5805 scope.go:122] "RemoveContainer" containerID="93bb816e462ef068a6af9188705606d14ac355598cfa13d717696e17d82e3bfd" Nov 03 18:17:05.351329 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:17:05.351308 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93bb816e462ef068a6af9188705606d14ac355598cfa13d717696e17d82e3bfd\": container with ID starting with 93bb816e462ef068a6af9188705606d14ac355598cfa13d717696e17d82e3bfd not found: ID does not exist" containerID="93bb816e462ef068a6af9188705606d14ac355598cfa13d717696e17d82e3bfd" Nov 03 18:17:05.351407 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:05.351336 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93bb816e462ef068a6af9188705606d14ac355598cfa13d717696e17d82e3bfd"} err="failed to get container status \"93bb816e462ef068a6af9188705606d14ac355598cfa13d717696e17d82e3bfd\": rpc error: code = NotFound desc = could not find container \"93bb816e462ef068a6af9188705606d14ac355598cfa13d717696e17d82e3bfd\": container with ID starting with 93bb816e462ef068a6af9188705606d14ac355598cfa13d717696e17d82e3bfd not found: ID does not exist" Nov 03 18:17:05.492589 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:05.492557 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz"] Nov 03 18:17:05.538464 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:05.538429 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f79c02/simulator-w2-f79c02-kserve-5fccb59878-j9hvz"] Nov 03 18:17:06.074267 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:06.074233 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:06.976049 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:06.976016 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" path="/var/lib/kubelet/pods/ad8cc84a-716a-44c7-a257-1f630c48e1b6/volumes" Nov 03 18:17:10.833694 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:10.833658 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:14.897040 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:14.896999 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:15.790333 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:15.790300 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:16.178102 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:16.178066 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:16.484794 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:16.484714 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:16.686632 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:16.686595 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:16.892595 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:16.892557 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:17.362149 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:17.362108 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:21.384300 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:21.384255 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:24.130041 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:24.130007 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:24.335608 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:24.335568 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:24.544885 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:24.544799 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:25.061190 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:25.061152 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:26.189745 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:26.189709 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:26.243793 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:17:26.243724 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:17:26.385683 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:26.385653 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:27.826688 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:27.826649 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:29.966910 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:29.966867 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5"] Nov 03 18:17:30.180748 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:30.180713 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:30.415573 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:30.415533 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:31.815105 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:31.815075 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:32.162288 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:32.162201 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:32.370545 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:32.370508 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:32.812588 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:32.812543 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:34.089292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:34.089253 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:34.288918 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:34.288887 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:34.492481 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:34.492389 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:36.275237 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:36.275203 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w"] Nov 03 18:17:36.382502 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:36.382467 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9"] Nov 03 18:17:36.703845 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:36.703807 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:36.907333 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:36.907301 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:37.111458 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:37.111428 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:39.428344 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:39.428304 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:39.631700 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:39.631668 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:39.722790 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:39.722708 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd"] Nov 03 18:17:39.722947 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:39.722916 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" podUID="86ead782-7e42-4a09-854e-e5e3aa485554" containerName="istio-proxy" containerID="cri-o://8466544c8d87f41b83fbc8a4ff089ded9351c944f132e03ab14111ff2472b1ab" gracePeriod=30 Nov 03 18:17:39.844338 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:39.844246 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:41.045637 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:41.045604 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:41.276045 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:41.275865 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w" podUID="d41d4083-b660-4c12-8b03-47159d59b941" containerName="payload-processing" containerID="cri-o://8d0bd81b01f51fd2fd02559dd33e87cde0e6d7e0a9bda422976d4715f46febb4" gracePeriod=25 Nov 03 18:17:41.383011 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:41.382972 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9" podUID="d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5" containerName="payload-pre-processing" containerID="cri-o://6e024dc9c05442738aa29126f0fb09f3bc93954387acc37a8b56878fea1e7877" gracePeriod=25 Nov 03 18:17:42.452556 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:42.452483 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:42.667669 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:42.667635 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:42.882798 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:42.882755 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:43.348470 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:43.348443 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:44.983527 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:44.983504 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:17:44.987747 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:44.987673 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" containerID="cri-o://2df83bd5bb7e52464d02cf50f16ddc1c2feac2bee8133bfab44fe7e43c511b5b" gracePeriod=45 Nov 03 18:17:45.099969 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.099940 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ead782-7e42-4a09-854e-e5e3aa485554-kube-api-access-5pjzm\" (UniqueName: \"kubernetes.io/projected/86ead782-7e42-4a09-854e-e5e3aa485554-kube-api-access-5pjzm\") pod \"86ead782-7e42-4a09-854e-e5e3aa485554\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " Nov 03 18:17:45.100143 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.099980 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-credential-socket\") pod \"86ead782-7e42-4a09-854e-e5e3aa485554\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " Nov 03 18:17:45.100143 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.100023 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ead782-7e42-4a09-854e-e5e3aa485554-istio-token\" (UniqueName: \"kubernetes.io/projected/86ead782-7e42-4a09-854e-e5e3aa485554-istio-token\") pod \"86ead782-7e42-4a09-854e-e5e3aa485554\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " Nov 03 18:17:45.100143 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.100050 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-workload-certs\") pod \"86ead782-7e42-4a09-854e-e5e3aa485554\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " Nov 03 18:17:45.100143 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.100070 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-istio-data\") pod \"86ead782-7e42-4a09-854e-e5e3aa485554\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " Nov 03 18:17:45.100143 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.100133 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-workload-socket\") pod \"86ead782-7e42-4a09-854e-e5e3aa485554\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " Nov 03 18:17:45.100448 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.100190 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-istio-envoy\") pod \"86ead782-7e42-4a09-854e-e5e3aa485554\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " Nov 03 18:17:45.100448 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.100259 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86ead782-7e42-4a09-854e-e5e3aa485554-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ead782-7e42-4a09-854e-e5e3aa485554-istio-podinfo\") pod \"86ead782-7e42-4a09-854e-e5e3aa485554\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " Nov 03 18:17:45.100448 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.100329 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86ead782-7e42-4a09-854e-e5e3aa485554-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ead782-7e42-4a09-854e-e5e3aa485554-istiod-ca-cert\") pod \"86ead782-7e42-4a09-854e-e5e3aa485554\" (UID: \"86ead782-7e42-4a09-854e-e5e3aa485554\") " Nov 03 18:17:45.100448 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.100329 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-credential-socket" pod "86ead782-7e42-4a09-854e-e5e3aa485554" (UID: "86ead782-7e42-4a09-854e-e5e3aa485554"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:17:45.100448 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.100386 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-istio-data" pod "86ead782-7e42-4a09-854e-e5e3aa485554" (UID: "86ead782-7e42-4a09-854e-e5e3aa485554"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:17:45.100728 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.100524 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-workload-socket" pod "86ead782-7e42-4a09-854e-e5e3aa485554" (UID: "86ead782-7e42-4a09-854e-e5e3aa485554"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:17:45.100830 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.100806 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-workload-certs" pod "86ead782-7e42-4a09-854e-e5e3aa485554" (UID: "86ead782-7e42-4a09-854e-e5e3aa485554"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:17:45.100894 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.100870 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ead782-7e42-4a09-854e-e5e3aa485554-istiod-ca-cert" pod "86ead782-7e42-4a09-854e-e5e3aa485554" (UID: "86ead782-7e42-4a09-854e-e5e3aa485554"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:17:45.100973 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.100953 5805 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-credential-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:45.101031 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.100980 5805 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-workload-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:45.101031 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.100996 5805 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-istio-data\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:45.101031 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.101009 5805 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-workload-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:45.102049 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.102029 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ead782-7e42-4a09-854e-e5e3aa485554-kube-api-access-5pjzm" pod "86ead782-7e42-4a09-854e-e5e3aa485554" (UID: "86ead782-7e42-4a09-854e-e5e3aa485554"). InnerVolumeSpecName "kube-api-access-5pjzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:17:45.102224 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.102209 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ead782-7e42-4a09-854e-e5e3aa485554-istio-token" pod "86ead782-7e42-4a09-854e-e5e3aa485554" (UID: "86ead782-7e42-4a09-854e-e5e3aa485554"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:17:45.102846 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.102822 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-istio-envoy" pod "86ead782-7e42-4a09-854e-e5e3aa485554" (UID: "86ead782-7e42-4a09-854e-e5e3aa485554"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:17:45.102942 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.102843 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86ead782-7e42-4a09-854e-e5e3aa485554-istio-podinfo" pod "86ead782-7e42-4a09-854e-e5e3aa485554" (UID: "86ead782-7e42-4a09-854e-e5e3aa485554"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:17:45.135346 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.135315 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": dial tcp 10.134.0.56:8000: connect: connection refused" Nov 03 18:17:45.201885 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.201849 5805 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ead782-7e42-4a09-854e-e5e3aa485554-istio-envoy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:45.201885 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.201876 5805 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ead782-7e42-4a09-854e-e5e3aa485554-istio-podinfo\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:45.201885 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.201885 5805 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ead782-7e42-4a09-854e-e5e3aa485554-istiod-ca-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:45.202164 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.201894 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pjzm\" (UniqueName: \"kubernetes.io/projected/86ead782-7e42-4a09-854e-e5e3aa485554-kube-api-access-5pjzm\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:45.202164 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.201905 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86ead782-7e42-4a09-854e-e5e3aa485554-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:45.240218 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.240164 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:17:45.302323 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.302289 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-model-cache\") pod \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\" (UID: \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\") " Nov 03 18:17:45.302490 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.302375 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce3c87ae-e083-48dd-b3f3-4d18b917c395-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce3c87ae-e083-48dd-b3f3-4d18b917c395-tls-certs\") pod \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\" (UID: \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\") " Nov 03 18:17:45.302490 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.302427 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-dshm\") pod \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\" (UID: \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\") " Nov 03 18:17:45.302490 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.302465 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-tmp-dir\") pod \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\" (UID: \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\") " Nov 03 18:17:45.302627 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.302495 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce3c87ae-e083-48dd-b3f3-4d18b917c395-kube-api-access-j9vgb\" (UniqueName: \"kubernetes.io/projected/ce3c87ae-e083-48dd-b3f3-4d18b917c395-kube-api-access-j9vgb\") pod \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\" (UID: \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\") " Nov 03 18:17:45.302627 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.302559 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-model-cache" pod "ce3c87ae-e083-48dd-b3f3-4d18b917c395" (UID: "ce3c87ae-e083-48dd-b3f3-4d18b917c395"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:17:45.302627 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.302568 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-home\" (UniqueName: \"kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-home\") pod \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\" (UID: \"ce3c87ae-e083-48dd-b3f3-4d18b917c395\") " Nov 03 18:17:45.302778 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.302759 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-home" pod "ce3c87ae-e083-48dd-b3f3-4d18b917c395" (UID: "ce3c87ae-e083-48dd-b3f3-4d18b917c395"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:17:45.302778 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.302769 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-tmp-dir" pod "ce3c87ae-e083-48dd-b3f3-4d18b917c395" (UID: "ce3c87ae-e083-48dd-b3f3-4d18b917c395"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:17:45.303023 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.303004 5805 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-home\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:45.303110 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.303024 5805 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-model-cache\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:45.303110 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.303035 5805 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-tmp-dir\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:45.304604 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.304585 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-dshm" pod "ce3c87ae-e083-48dd-b3f3-4d18b917c395" (UID: "ce3c87ae-e083-48dd-b3f3-4d18b917c395"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:17:45.304706 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.304669 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce3c87ae-e083-48dd-b3f3-4d18b917c395-tls-certs" pod "ce3c87ae-e083-48dd-b3f3-4d18b917c395" (UID: "ce3c87ae-e083-48dd-b3f3-4d18b917c395"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:17:45.304706 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.304695 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3c87ae-e083-48dd-b3f3-4d18b917c395-kube-api-access-j9vgb" pod "ce3c87ae-e083-48dd-b3f3-4d18b917c395" (UID: "ce3c87ae-e083-48dd-b3f3-4d18b917c395"). InnerVolumeSpecName "kube-api-access-j9vgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:17:45.404080 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.404041 5805 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce3c87ae-e083-48dd-b3f3-4d18b917c395-tls-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:45.404294 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.404076 5805 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce3c87ae-e083-48dd-b3f3-4d18b917c395-dshm\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:45.404294 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.404173 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9vgb\" (UniqueName: \"kubernetes.io/projected/ce3c87ae-e083-48dd-b3f3-4d18b917c395-kube-api-access-j9vgb\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:45.567327 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.567295 5805 generic.go:356] "Generic (PLEG): container finished" podID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerID="2df83bd5bb7e52464d02cf50f16ddc1c2feac2bee8133bfab44fe7e43c511b5b" exitCode=0 Nov 03 18:17:45.567542 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.567352 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" Nov 03 18:17:45.567542 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.567379 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" event={"ID":"ce3c87ae-e083-48dd-b3f3-4d18b917c395","Type":"ContainerDied","Data":"2df83bd5bb7e52464d02cf50f16ddc1c2feac2bee8133bfab44fe7e43c511b5b"} Nov 03 18:17:45.567542 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.567463 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5" event={"ID":"ce3c87ae-e083-48dd-b3f3-4d18b917c395","Type":"ContainerDied","Data":"ef9783f9870367d4425b866551b974635660074680b7d13f4745783c883a136b"} Nov 03 18:17:45.567542 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.567489 5805 scope.go:122] "RemoveContainer" containerID="2df83bd5bb7e52464d02cf50f16ddc1c2feac2bee8133bfab44fe7e43c511b5b" Nov 03 18:17:45.569192 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.569169 5805 generic.go:356] "Generic (PLEG): container finished" podID="86ead782-7e42-4a09-854e-e5e3aa485554" containerID="8466544c8d87f41b83fbc8a4ff089ded9351c944f132e03ab14111ff2472b1ab" exitCode=0 Nov 03 18:17:45.569304 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.569205 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" Nov 03 18:17:45.569304 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.569263 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" event={"ID":"86ead782-7e42-4a09-854e-e5e3aa485554","Type":"ContainerDied","Data":"8466544c8d87f41b83fbc8a4ff089ded9351c944f132e03ab14111ff2472b1ab"} Nov 03 18:17:45.569421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.569322 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd" event={"ID":"86ead782-7e42-4a09-854e-e5e3aa485554","Type":"ContainerDied","Data":"12e8fbbf4695f6b1eee2a4afea659e5b90054d36e3b72fe714cf017c88c8004e"} Nov 03 18:17:45.577595 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.577575 5805 scope.go:122] "RemoveContainer" containerID="2df83bd5bb7e52464d02cf50f16ddc1c2feac2bee8133bfab44fe7e43c511b5b" Nov 03 18:17:45.577931 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:17:45.577910 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2df83bd5bb7e52464d02cf50f16ddc1c2feac2bee8133bfab44fe7e43c511b5b\": container with ID starting with 2df83bd5bb7e52464d02cf50f16ddc1c2feac2bee8133bfab44fe7e43c511b5b not found: ID does not exist" containerID="2df83bd5bb7e52464d02cf50f16ddc1c2feac2bee8133bfab44fe7e43c511b5b" Nov 03 18:17:45.578017 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.577943 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2df83bd5bb7e52464d02cf50f16ddc1c2feac2bee8133bfab44fe7e43c511b5b"} err="failed to get container status \"2df83bd5bb7e52464d02cf50f16ddc1c2feac2bee8133bfab44fe7e43c511b5b\": rpc error: code = NotFound desc = could not find container \"2df83bd5bb7e52464d02cf50f16ddc1c2feac2bee8133bfab44fe7e43c511b5b\": container with ID starting with 2df83bd5bb7e52464d02cf50f16ddc1c2feac2bee8133bfab44fe7e43c511b5b not found: ID does not exist" Nov 03 18:17:45.578017 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.577964 5805 scope.go:122] "RemoveContainer" containerID="8466544c8d87f41b83fbc8a4ff089ded9351c944f132e03ab14111ff2472b1ab" Nov 03 18:17:45.586828 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.586811 5805 scope.go:122] "RemoveContainer" containerID="8466544c8d87f41b83fbc8a4ff089ded9351c944f132e03ab14111ff2472b1ab" Nov 03 18:17:45.587076 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:17:45.587053 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8466544c8d87f41b83fbc8a4ff089ded9351c944f132e03ab14111ff2472b1ab\": container with ID starting with 8466544c8d87f41b83fbc8a4ff089ded9351c944f132e03ab14111ff2472b1ab not found: ID does not exist" containerID="8466544c8d87f41b83fbc8a4ff089ded9351c944f132e03ab14111ff2472b1ab" Nov 03 18:17:45.587143 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.587085 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8466544c8d87f41b83fbc8a4ff089ded9351c944f132e03ab14111ff2472b1ab"} err="failed to get container status \"8466544c8d87f41b83fbc8a4ff089ded9351c944f132e03ab14111ff2472b1ab\": rpc error: code = NotFound desc = could not find container \"8466544c8d87f41b83fbc8a4ff089ded9351c944f132e03ab14111ff2472b1ab\": container with ID starting with 8466544c8d87f41b83fbc8a4ff089ded9351c944f132e03ab14111ff2472b1ab not found: ID does not exist" Nov 03 18:17:45.721764 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.721733 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd"] Nov 03 18:17:45.767537 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.767505 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f79c02-openshift-default-8659465687-957dd"] Nov 03 18:17:45.830647 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.830566 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5"] Nov 03 18:17:45.836604 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:45.836576 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-2b4f46/distinct-w5-2b4f46-kserve-86bf75ffd4-wdqx5"] Nov 03 18:17:46.975732 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:46.975702 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86ead782-7e42-4a09-854e-e5e3aa485554" path="/var/lib/kubelet/pods/86ead782-7e42-4a09-854e-e5e3aa485554/volumes" Nov 03 18:17:46.976116 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:46.976102 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" path="/var/lib/kubelet/pods/ce3c87ae-e083-48dd-b3f3-4d18b917c395/volumes" Nov 03 18:17:47.375532 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:47.375491 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:48.476310 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:48.476278 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:49.982160 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:49.982121 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9" podUID="d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:17:49.983639 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:49.983606 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w" podUID="d41d4083-b660-4c12-8b03-47159d59b941" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:17:51.297067 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:51.297033 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42"] Nov 03 18:17:51.297454 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:51.297236 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" podUID="c5e3b992-04fc-4c47-b353-55a0fa75f7ed" containerName="maas-api" containerID="cri-o://d0022d04c9948338905d71a5635142e39b84b7acc316113e3c6f932443be5882" gracePeriod=30 Nov 03 18:17:51.447339 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:51.447297 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:51.577474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:51.575729 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" Nov 03 18:17:51.604852 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:51.604820 5805 generic.go:356] "Generic (PLEG): container finished" podID="c5e3b992-04fc-4c47-b353-55a0fa75f7ed" containerID="d0022d04c9948338905d71a5635142e39b84b7acc316113e3c6f932443be5882" exitCode=0 Nov 03 18:17:51.604976 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:51.604857 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" Nov 03 18:17:51.604976 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:51.604905 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" event={"ID":"c5e3b992-04fc-4c47-b353-55a0fa75f7ed","Type":"ContainerDied","Data":"d0022d04c9948338905d71a5635142e39b84b7acc316113e3c6f932443be5882"} Nov 03 18:17:51.605066 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:51.604980 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42" event={"ID":"c5e3b992-04fc-4c47-b353-55a0fa75f7ed","Type":"ContainerDied","Data":"f8252cd1eaf631222d3ef53e047e10fe81f081c1316111d130845b23904aef7d"} Nov 03 18:17:51.605066 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:51.605002 5805 scope.go:122] "RemoveContainer" containerID="d0022d04c9948338905d71a5635142e39b84b7acc316113e3c6f932443be5882" Nov 03 18:17:51.613985 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:51.613964 5805 scope.go:122] "RemoveContainer" containerID="d0022d04c9948338905d71a5635142e39b84b7acc316113e3c6f932443be5882" Nov 03 18:17:51.614220 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:17:51.614203 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0022d04c9948338905d71a5635142e39b84b7acc316113e3c6f932443be5882\": container with ID starting with d0022d04c9948338905d71a5635142e39b84b7acc316113e3c6f932443be5882 not found: ID does not exist" containerID="d0022d04c9948338905d71a5635142e39b84b7acc316113e3c6f932443be5882" Nov 03 18:17:51.614273 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:51.614228 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0022d04c9948338905d71a5635142e39b84b7acc316113e3c6f932443be5882"} err="failed to get container status \"d0022d04c9948338905d71a5635142e39b84b7acc316113e3c6f932443be5882\": rpc error: code = NotFound desc = could not find container \"d0022d04c9948338905d71a5635142e39b84b7acc316113e3c6f932443be5882\": container with ID starting with d0022d04c9948338905d71a5635142e39b84b7acc316113e3c6f932443be5882 not found: ID does not exist" Nov 03 18:17:51.673445 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:51.673379 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-metrics-certs\") pod \"c5e3b992-04fc-4c47-b353-55a0fa75f7ed\" (UID: \"c5e3b992-04fc-4c47-b353-55a0fa75f7ed\") " Nov 03 18:17:51.673445 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:51.673432 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-maas-api-tls\") pod \"c5e3b992-04fc-4c47-b353-55a0fa75f7ed\" (UID: \"c5e3b992-04fc-4c47-b353-55a0fa75f7ed\") " Nov 03 18:17:51.673715 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:51.673615 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-kube-api-access-49tkb\" (UniqueName: \"kubernetes.io/projected/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-kube-api-access-49tkb\") pod \"c5e3b992-04fc-4c47-b353-55a0fa75f7ed\" (UID: \"c5e3b992-04fc-4c47-b353-55a0fa75f7ed\") " Nov 03 18:17:51.673715 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:51.673686 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-openshift-service-ca\") pod \"c5e3b992-04fc-4c47-b353-55a0fa75f7ed\" (UID: \"c5e3b992-04fc-4c47-b353-55a0fa75f7ed\") " Nov 03 18:17:51.674345 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:51.674306 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-openshift-service-ca" pod "c5e3b992-04fc-4c47-b353-55a0fa75f7ed" (UID: "c5e3b992-04fc-4c47-b353-55a0fa75f7ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:17:51.675950 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:51.675921 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-metrics-certs" pod "c5e3b992-04fc-4c47-b353-55a0fa75f7ed" (UID: "c5e3b992-04fc-4c47-b353-55a0fa75f7ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:17:51.676077 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:51.676060 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-maas-api-tls" pod "c5e3b992-04fc-4c47-b353-55a0fa75f7ed" (UID: "c5e3b992-04fc-4c47-b353-55a0fa75f7ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:17:51.676246 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:51.676227 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-kube-api-access-49tkb" pod "c5e3b992-04fc-4c47-b353-55a0fa75f7ed" (UID: "c5e3b992-04fc-4c47-b353-55a0fa75f7ed"). InnerVolumeSpecName "kube-api-access-49tkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:17:51.775122 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:51.775091 5805 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-metrics-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:51.775122 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:51.775117 5805 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-maas-api-tls\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:51.775122 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:51.775127 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49tkb\" (UniqueName: \"kubernetes.io/projected/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-kube-api-access-49tkb\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:51.775338 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:51.775137 5805 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e3b992-04fc-4c47-b353-55a0fa75f7ed-openshift-service-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:17:52.019539 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:52.019505 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42"] Nov 03 18:17:52.025537 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:52.025511 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-2b4f46-565bdd7788-5kw42"] Nov 03 18:17:52.864702 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:52.864670 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:52.975113 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:52.975083 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5e3b992-04fc-4c47-b353-55a0fa75f7ed" path="/var/lib/kubelet/pods/c5e3b992-04fc-4c47-b353-55a0fa75f7ed/volumes" Nov 03 18:17:53.070236 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:53.070207 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:53.269582 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:53.269510 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:53.763061 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:53.763021 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:53.967152 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:53.967104 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:54.163660 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:54.163628 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:56.234449 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:17:56.234415 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:17:57.861965 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:57.861913 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:17:59.975017 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:59.974977 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w" podUID="d41d4083-b660-4c12-8b03-47159d59b941" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:17:59.975420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:17:59.974983 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9" podUID="d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:18:00.388801 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:00.388768 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:00.587316 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:00.587281 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:00.690947 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:00.690865 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw"] Nov 03 18:18:00.691132 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:00.691107 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" podUID="35c5ded9-15bd-4f4f-b09e-0e91231efd07" containerName="istio-proxy" containerID="cri-o://77dbbde3602bfd27bb3cec51e53d7d4559bdf9d3e6bbeea8d18c43f18a7d5671" gracePeriod=30 Nov 03 18:18:00.788994 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:00.788961 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:01.524728 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.524704 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w" Nov 03 18:18:01.629650 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.629626 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9" Nov 03 18:18:01.670762 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.670729 5805 generic.go:356] "Generic (PLEG): container finished" podID="d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5" containerID="6e024dc9c05442738aa29126f0fb09f3bc93954387acc37a8b56878fea1e7877" exitCode=0 Nov 03 18:18:01.670926 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.670767 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9" Nov 03 18:18:01.670926 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.670811 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9" event={"ID":"d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5","Type":"ContainerDied","Data":"6e024dc9c05442738aa29126f0fb09f3bc93954387acc37a8b56878fea1e7877"} Nov 03 18:18:01.670926 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.670868 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9" event={"ID":"d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5","Type":"ContainerDied","Data":"e02bb38b8b924a2d688915b14de9ee32cd978b772cbc2a100032cd472e695472"} Nov 03 18:18:01.670926 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.670901 5805 scope.go:122] "RemoveContainer" containerID="6e024dc9c05442738aa29126f0fb09f3bc93954387acc37a8b56878fea1e7877" Nov 03 18:18:01.672253 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.672234 5805 generic.go:356] "Generic (PLEG): container finished" podID="d41d4083-b660-4c12-8b03-47159d59b941" containerID="8d0bd81b01f51fd2fd02559dd33e87cde0e6d7e0a9bda422976d4715f46febb4" exitCode=0 Nov 03 18:18:01.672376 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.672277 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w" Nov 03 18:18:01.672376 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.672340 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w" event={"ID":"d41d4083-b660-4c12-8b03-47159d59b941","Type":"ContainerDied","Data":"8d0bd81b01f51fd2fd02559dd33e87cde0e6d7e0a9bda422976d4715f46febb4"} Nov 03 18:18:01.672551 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.672379 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w" event={"ID":"d41d4083-b660-4c12-8b03-47159d59b941","Type":"ContainerDied","Data":"2b6e46332b24e055a273edf0b428e7147231a4bdb3a38ac04b3ecf56d318f36e"} Nov 03 18:18:01.677824 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.677802 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d41d4083-b660-4c12-8b03-47159d59b941-kube-api-access-nkdj2\" (UniqueName: \"kubernetes.io/projected/d41d4083-b660-4c12-8b03-47159d59b941-kube-api-access-nkdj2\") pod \"d41d4083-b660-4c12-8b03-47159d59b941\" (UID: \"d41d4083-b660-4c12-8b03-47159d59b941\") " Nov 03 18:18:01.677929 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.677840 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d41d4083-b660-4c12-8b03-47159d59b941-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d41d4083-b660-4c12-8b03-47159d59b941-plugins-config-volume\") pod \"d41d4083-b660-4c12-8b03-47159d59b941\" (UID: \"d41d4083-b660-4c12-8b03-47159d59b941\") " Nov 03 18:18:01.678244 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.678208 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d41d4083-b660-4c12-8b03-47159d59b941-plugins-config-volume" pod "d41d4083-b660-4c12-8b03-47159d59b941" (UID: "d41d4083-b660-4c12-8b03-47159d59b941"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:18:01.679802 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.679777 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d41d4083-b660-4c12-8b03-47159d59b941-kube-api-access-nkdj2" pod "d41d4083-b660-4c12-8b03-47159d59b941" (UID: "d41d4083-b660-4c12-8b03-47159d59b941"). InnerVolumeSpecName "kube-api-access-nkdj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:18:01.680254 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.680230 5805 scope.go:122] "RemoveContainer" containerID="6e024dc9c05442738aa29126f0fb09f3bc93954387acc37a8b56878fea1e7877" Nov 03 18:18:01.680635 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:18:01.680619 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e024dc9c05442738aa29126f0fb09f3bc93954387acc37a8b56878fea1e7877\": container with ID starting with 6e024dc9c05442738aa29126f0fb09f3bc93954387acc37a8b56878fea1e7877 not found: ID does not exist" containerID="6e024dc9c05442738aa29126f0fb09f3bc93954387acc37a8b56878fea1e7877" Nov 03 18:18:01.680683 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.680650 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e024dc9c05442738aa29126f0fb09f3bc93954387acc37a8b56878fea1e7877"} err="failed to get container status \"6e024dc9c05442738aa29126f0fb09f3bc93954387acc37a8b56878fea1e7877\": rpc error: code = NotFound desc = could not find container \"6e024dc9c05442738aa29126f0fb09f3bc93954387acc37a8b56878fea1e7877\": container with ID starting with 6e024dc9c05442738aa29126f0fb09f3bc93954387acc37a8b56878fea1e7877 not found: ID does not exist" Nov 03 18:18:01.680683 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.680666 5805 scope.go:122] "RemoveContainer" containerID="8d0bd81b01f51fd2fd02559dd33e87cde0e6d7e0a9bda422976d4715f46febb4" Nov 03 18:18:01.688707 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.688690 5805 scope.go:122] "RemoveContainer" containerID="8d0bd81b01f51fd2fd02559dd33e87cde0e6d7e0a9bda422976d4715f46febb4" Nov 03 18:18:01.688986 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:18:01.688963 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d0bd81b01f51fd2fd02559dd33e87cde0e6d7e0a9bda422976d4715f46febb4\": container with ID starting with 8d0bd81b01f51fd2fd02559dd33e87cde0e6d7e0a9bda422976d4715f46febb4 not found: ID does not exist" containerID="8d0bd81b01f51fd2fd02559dd33e87cde0e6d7e0a9bda422976d4715f46febb4" Nov 03 18:18:01.689035 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.688994 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d0bd81b01f51fd2fd02559dd33e87cde0e6d7e0a9bda422976d4715f46febb4"} err="failed to get container status \"8d0bd81b01f51fd2fd02559dd33e87cde0e6d7e0a9bda422976d4715f46febb4\": rpc error: code = NotFound desc = could not find container \"8d0bd81b01f51fd2fd02559dd33e87cde0e6d7e0a9bda422976d4715f46febb4\": container with ID starting with 8d0bd81b01f51fd2fd02559dd33e87cde0e6d7e0a9bda422976d4715f46febb4 not found: ID does not exist" Nov 03 18:18:01.778699 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.778674 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5-plugins-config-volume\") pod \"d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5\" (UID: \"d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5\") " Nov 03 18:18:01.778808 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.778781 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5-kube-api-access-vfkjk\" (UniqueName: \"kubernetes.io/projected/d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5-kube-api-access-vfkjk\") pod \"d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5\" (UID: \"d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5\") " Nov 03 18:18:01.779003 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.778982 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5-plugins-config-volume" pod "d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5" (UID: "d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:18:01.779102 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.779089 5805 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5-plugins-config-volume\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:18:01.779147 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.779105 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkdj2\" (UniqueName: \"kubernetes.io/projected/d41d4083-b660-4c12-8b03-47159d59b941-kube-api-access-nkdj2\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:18:01.779147 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.779114 5805 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d41d4083-b660-4c12-8b03-47159d59b941-plugins-config-volume\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:18:01.780620 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.780601 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5-kube-api-access-vfkjk" pod "d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5" (UID: "d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5"). InnerVolumeSpecName "kube-api-access-vfkjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:18:01.880193 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:01.880153 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfkjk\" (UniqueName: \"kubernetes.io/projected/d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5-kube-api-access-vfkjk\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:18:02.087378 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:02.087351 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9"] Nov 03 18:18:02.092762 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:02.092739 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2b4f46-79549884f5-58dl9"] Nov 03 18:18:02.194930 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:02.194850 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w"] Nov 03 18:18:02.201640 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:02.201615 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-2b4f46-78c48c587c-snr7w"] Nov 03 18:18:02.479493 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:02.479411 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:02.975598 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:02.975567 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d41d4083-b660-4c12-8b03-47159d59b941" path="/var/lib/kubelet/pods/d41d4083-b660-4c12-8b03-47159d59b941/volumes" Nov 03 18:18:02.975987 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:02.975974 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5" path="/var/lib/kubelet/pods/d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5/volumes" Nov 03 18:18:05.199019 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:05.198982 5805 patch_prober.go:29] interesting pod/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:18:05.199414 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:05.199042 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" podUID="35c5ded9-15bd-4f4f-b09e-0e91231efd07" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:18:05.945749 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:05.945720 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:18:06.124975 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.124944 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c5ded9-15bd-4f4f-b09e-0e91231efd07-kube-api-access-22nvw\" (UniqueName: \"kubernetes.io/projected/35c5ded9-15bd-4f4f-b09e-0e91231efd07-kube-api-access-22nvw\") pod \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " Nov 03 18:18:06.125140 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.125005 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-credential-socket\") pod \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " Nov 03 18:18:06.125140 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.125052 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-podinfo\") pod \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " Nov 03 18:18:06.125140 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.125102 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-data\") pod \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " Nov 03 18:18:06.125365 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.125185 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-token\" (UniqueName: \"kubernetes.io/projected/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-token\") pod \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " Nov 03 18:18:06.125365 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.125245 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istiod-ca-cert\") pod \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " Nov 03 18:18:06.125365 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.125295 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-workload-socket\") pod \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " Nov 03 18:18:06.125365 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.125271 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-credential-socket" pod "35c5ded9-15bd-4f4f-b09e-0e91231efd07" (UID: "35c5ded9-15bd-4f4f-b09e-0e91231efd07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:18:06.125581 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.125362 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-workload-certs\") pod \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " Nov 03 18:18:06.125581 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.125515 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-envoy\") pod \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\" (UID: \"35c5ded9-15bd-4f4f-b09e-0e91231efd07\") " Nov 03 18:18:06.125788 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.125514 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-workload-socket" pod "35c5ded9-15bd-4f4f-b09e-0e91231efd07" (UID: "35c5ded9-15bd-4f4f-b09e-0e91231efd07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:18:06.125788 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.125763 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istiod-ca-cert" pod "35c5ded9-15bd-4f4f-b09e-0e91231efd07" (UID: "35c5ded9-15bd-4f4f-b09e-0e91231efd07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:18:06.125788 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.125686 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-workload-certs" pod "35c5ded9-15bd-4f4f-b09e-0e91231efd07" (UID: "35c5ded9-15bd-4f4f-b09e-0e91231efd07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:18:06.125788 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.125781 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-data" pod "35c5ded9-15bd-4f4f-b09e-0e91231efd07" (UID: "35c5ded9-15bd-4f4f-b09e-0e91231efd07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:18:06.126058 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.126015 5805 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-workload-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:18:06.126058 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.126039 5805 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-workload-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:18:06.126058 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.126054 5805 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-credential-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:18:06.126221 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.126070 5805 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-data\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:18:06.126221 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.126085 5805 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istiod-ca-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:18:06.127906 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.127855 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-token" pod "35c5ded9-15bd-4f4f-b09e-0e91231efd07" (UID: "35c5ded9-15bd-4f4f-b09e-0e91231efd07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:18:06.128024 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.127961 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c5ded9-15bd-4f4f-b09e-0e91231efd07-kube-api-access-22nvw" pod "35c5ded9-15bd-4f4f-b09e-0e91231efd07" (UID: "35c5ded9-15bd-4f4f-b09e-0e91231efd07"). InnerVolumeSpecName "kube-api-access-22nvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:18:06.128024 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.127977 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-envoy" pod "35c5ded9-15bd-4f4f-b09e-0e91231efd07" (UID: "35c5ded9-15bd-4f4f-b09e-0e91231efd07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:18:06.128108 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.128044 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-podinfo" pod "35c5ded9-15bd-4f4f-b09e-0e91231efd07" (UID: "35c5ded9-15bd-4f4f-b09e-0e91231efd07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:18:06.227608 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.227574 5805 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-podinfo\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:18:06.227608 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.227603 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:18:06.228002 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.227618 5805 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c5ded9-15bd-4f4f-b09e-0e91231efd07-istio-envoy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:18:06.228002 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.227631 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22nvw\" (UniqueName: \"kubernetes.io/projected/35c5ded9-15bd-4f4f-b09e-0e91231efd07-kube-api-access-22nvw\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:18:06.705957 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.705916 5805 generic.go:356] "Generic (PLEG): container finished" podID="35c5ded9-15bd-4f4f-b09e-0e91231efd07" containerID="77dbbde3602bfd27bb3cec51e53d7d4559bdf9d3e6bbeea8d18c43f18a7d5671" exitCode=0 Nov 03 18:18:06.706145 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.705975 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" Nov 03 18:18:06.706145 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.706003 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" event={"ID":"35c5ded9-15bd-4f4f-b09e-0e91231efd07","Type":"ContainerDied","Data":"77dbbde3602bfd27bb3cec51e53d7d4559bdf9d3e6bbeea8d18c43f18a7d5671"} Nov 03 18:18:06.706145 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.706073 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw" event={"ID":"35c5ded9-15bd-4f4f-b09e-0e91231efd07","Type":"ContainerDied","Data":"04598f730b1a3ec2e952715aa99cdaf431274686814a184d5a6418829b8b90b5"} Nov 03 18:18:06.706145 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.706104 5805 scope.go:122] "RemoveContainer" containerID="77dbbde3602bfd27bb3cec51e53d7d4559bdf9d3e6bbeea8d18c43f18a7d5671" Nov 03 18:18:06.718434 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.718410 5805 scope.go:122] "RemoveContainer" containerID="77dbbde3602bfd27bb3cec51e53d7d4559bdf9d3e6bbeea8d18c43f18a7d5671" Nov 03 18:18:06.718769 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:18:06.718745 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77dbbde3602bfd27bb3cec51e53d7d4559bdf9d3e6bbeea8d18c43f18a7d5671\": container with ID starting with 77dbbde3602bfd27bb3cec51e53d7d4559bdf9d3e6bbeea8d18c43f18a7d5671 not found: ID does not exist" containerID="77dbbde3602bfd27bb3cec51e53d7d4559bdf9d3e6bbeea8d18c43f18a7d5671" Nov 03 18:18:06.718850 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.718782 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77dbbde3602bfd27bb3cec51e53d7d4559bdf9d3e6bbeea8d18c43f18a7d5671"} err="failed to get container status \"77dbbde3602bfd27bb3cec51e53d7d4559bdf9d3e6bbeea8d18c43f18a7d5671\": rpc error: code = NotFound desc = could not find container \"77dbbde3602bfd27bb3cec51e53d7d4559bdf9d3e6bbeea8d18c43f18a7d5671\": container with ID starting with 77dbbde3602bfd27bb3cec51e53d7d4559bdf9d3e6bbeea8d18c43f18a7d5671 not found: ID does not exist" Nov 03 18:18:06.770627 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.770597 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:06.860359 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.860330 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw"] Nov 03 18:18:06.889510 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.889486 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2b4f46-openshift-default-588d6bd46c-xz9sw"] Nov 03 18:18:06.971406 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.971305 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:06.977471 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:06.977443 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35c5ded9-15bd-4f4f-b09e-0e91231efd07" path="/var/lib/kubelet/pods/35c5ded9-15bd-4f4f-b09e-0e91231efd07/volumes" Nov 03 18:18:08.296725 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:08.296688 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:11.472183 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:11.472122 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:15.068804 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:15.068773 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:15.959591 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:15.959551 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:16.152271 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:16.152243 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:20.482837 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:20.482802 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:25.005585 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:25.005544 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:25.531491 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:25.531447 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:26.020439 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:26.020390 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:26.234841 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:18:26.234799 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:18:29.559451 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:29.559417 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:30.601118 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:30.601086 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:30.809379 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:30.809345 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:31.783844 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:31.783806 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:34.063067 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:34.063031 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:37.506159 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:37.506123 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:37.707729 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:37.707689 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:37.912245 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:37.912095 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:38.571802 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:38.571751 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:43.066178 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:43.066140 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:45.525420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:45.524500 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vgzph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:18:45.525420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:45.524567 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" podUID="d685b092-b52b-4ca9-a114-dd71a3be6f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:18:47.762085 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:47.762047 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:49.692314 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:49.692278 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:52.357308 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:52.357274 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:18:56.235177 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:18:56.235143 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:18:56.873879 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:18:56.873842 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:19:01.372512 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:01.372475 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:19:05.925587 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:05.925509 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:19:10.365543 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:10.365510 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:19:14.962258 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:14.962182 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:19:16.511694 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:16.511659 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:19:19.461489 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:19.461446 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:19:23.964258 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:23.964218 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:19:26.234709 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:19:26.234673 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:19:26.644655 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:26.644618 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:19:28.604327 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:28.604294 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:19:31.551331 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:31.551200 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf"] Nov 03 18:19:31.596306 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:31.596266 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l"] Nov 03 18:19:31.702131 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:31.702095 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:19:33.362875 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:33.362838 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:19:37.895860 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:37.895824 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:19:39.081330 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:39.081259 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:19:39.285747 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:39.285715 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:19:41.869330 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.869292 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13943829-jl86x"] Nov 03 18:19:41.869779 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.869745 5805 cpu_manager.go:414] "containerMap: removing container" podUID="c5e3b992-04fc-4c47-b353-55a0fa75f7ed" containerName="maas-api" Nov 03 18:19:41.869779 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.869760 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5e3b992-04fc-4c47-b353-55a0fa75f7ed" containerName="maas-api" Nov 03 18:19:41.869779 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.869776 5805 cpu_manager.go:414] "containerMap: removing container" podUID="d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5" containerName="payload-pre-processing" Nov 03 18:19:41.869921 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.869783 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5" containerName="payload-pre-processing" Nov 03 18:19:41.869921 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.869808 5805 cpu_manager.go:414] "containerMap: removing container" podUID="d41d4083-b660-4c12-8b03-47159d59b941" containerName="payload-processing" Nov 03 18:19:41.869921 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.869815 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d41d4083-b660-4c12-8b03-47159d59b941" containerName="payload-processing" Nov 03 18:19:41.869921 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.869828 5805 cpu_manager.go:414] "containerMap: removing container" podUID="86ead782-7e42-4a09-854e-e5e3aa485554" containerName="istio-proxy" Nov 03 18:19:41.869921 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.869836 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ead782-7e42-4a09-854e-e5e3aa485554" containerName="istio-proxy" Nov 03 18:19:41.869921 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.869850 5805 cpu_manager.go:414] "containerMap: removing container" podUID="e4120351-f702-437b-aa9b-40979fa30455" containerName="istio-proxy" Nov 03 18:19:41.869921 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.869857 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4120351-f702-437b-aa9b-40979fa30455" containerName="istio-proxy" Nov 03 18:19:41.869921 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.869870 5805 cpu_manager.go:414] "containerMap: removing container" podUID="73900785-3360-4c42-8a5d-8c0e258fc637" containerName="cleanup" Nov 03 18:19:41.869921 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.869877 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73900785-3360-4c42-8a5d-8c0e258fc637" containerName="cleanup" Nov 03 18:19:41.869921 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.869888 5805 cpu_manager.go:414] "containerMap: removing container" podUID="8664b8f5-f2eb-4d49-84d6-d568be08e781" containerName="revoke-keys" Nov 03 18:19:41.869921 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.869896 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8664b8f5-f2eb-4d49-84d6-d568be08e781" containerName="revoke-keys" Nov 03 18:19:41.869921 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.869908 5805 cpu_manager.go:414] "containerMap: removing container" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" Nov 03 18:19:41.869921 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.869915 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" Nov 03 18:19:41.869921 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.869927 5805 cpu_manager.go:414] "containerMap: removing container" podUID="35c5ded9-15bd-4f4f-b09e-0e91231efd07" containerName="istio-proxy" Nov 03 18:19:41.870587 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.869935 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35c5ded9-15bd-4f4f-b09e-0e91231efd07" containerName="istio-proxy" Nov 03 18:19:41.870587 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.869948 5805 cpu_manager.go:414] "containerMap: removing container" podUID="94ef80bb-de9b-4163-9db7-0fd6f19a8eca" containerName="istio-proxy" Nov 03 18:19:41.870587 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.869955 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94ef80bb-de9b-4163-9db7-0fd6f19a8eca" containerName="istio-proxy" Nov 03 18:19:41.870587 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.869972 5805 cpu_manager.go:414] "containerMap: removing container" podUID="b78a7ba2-1042-492c-b52c-dca73afd27d5" containerName="cleanup" Nov 03 18:19:41.870587 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.869979 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b78a7ba2-1042-492c-b52c-dca73afd27d5" containerName="cleanup" Nov 03 18:19:41.870587 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.870003 5805 cpu_manager.go:414] "containerMap: removing container" podUID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" containerName="main" Nov 03 18:19:41.870587 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.870010 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" containerName="main" Nov 03 18:19:41.870587 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.870020 5805 cpu_manager.go:414] "containerMap: removing container" podUID="36b2f655-b26e-4a96-a114-5596317d889e" containerName="payload-pre-processing" Nov 03 18:19:41.870587 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.870027 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b2f655-b26e-4a96-a114-5596317d889e" containerName="payload-pre-processing" Nov 03 18:19:41.870587 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.870112 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5cd7c49-cbc5-404d-bfb2-ef58348f7ae5" containerName="payload-pre-processing" Nov 03 18:19:41.870587 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.870123 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="d41d4083-b660-4c12-8b03-47159d59b941" containerName="payload-processing" Nov 03 18:19:41.870587 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.870129 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ead782-7e42-4a09-854e-e5e3aa485554" containerName="istio-proxy" Nov 03 18:19:41.870587 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.870136 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4120351-f702-437b-aa9b-40979fa30455" containerName="istio-proxy" Nov 03 18:19:41.870587 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.870142 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="8664b8f5-f2eb-4d49-84d6-d568be08e781" containerName="revoke-keys" Nov 03 18:19:41.870587 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.870148 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce3c87ae-e083-48dd-b3f3-4d18b917c395" containerName="main" Nov 03 18:19:41.870587 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.870152 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="35c5ded9-15bd-4f4f-b09e-0e91231efd07" containerName="istio-proxy" Nov 03 18:19:41.870587 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.870159 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="94ef80bb-de9b-4163-9db7-0fd6f19a8eca" containerName="istio-proxy" Nov 03 18:19:41.870587 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.870166 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad8cc84a-716a-44c7-a257-1f630c48e1b6" containerName="main" Nov 03 18:19:41.870587 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.870172 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b2f655-b26e-4a96-a114-5596317d889e" containerName="payload-pre-processing" Nov 03 18:19:41.870587 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.870178 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="d22875d5-06bf-4abe-9961-9f5c9922eac0" containerName="cleanup" Nov 03 18:19:41.870587 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.870186 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5e3b992-04fc-4c47-b353-55a0fa75f7ed" containerName="maas-api" Nov 03 18:19:41.873564 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.873539 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13943829-jl86x" Nov 03 18:19:41.982295 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.982254 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:19:41.982479 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.982314 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ndrlv\"" Nov 03 18:19:41.984403 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:41.984372 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13943829-jl86x"] Nov 03 18:19:42.015889 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:42.015864 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7238eabc-b932-49ce-aece-f9498937ecad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13943829-jl86x\" (UID: \"7238eabc-b932-49ce-aece-f9498937ecad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13943829-jl86x" Nov 03 18:19:42.016000 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:42.015919 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm9mp\" (UniqueName: \"kubernetes.io/projected/7238eabc-b932-49ce-aece-f9498937ecad-kube-api-access-mm9mp\") pod \"maas-api-revoke-sub-13943829-jl86x\" (UID: \"7238eabc-b932-49ce-aece-f9498937ecad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13943829-jl86x" Nov 03 18:19:42.016041 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:42.015993 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7238eabc-b932-49ce-aece-f9498937ecad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13943829-jl86x\" (UID: \"7238eabc-b932-49ce-aece-f9498937ecad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13943829-jl86x" Nov 03 18:19:42.116776 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:42.116746 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7238eabc-b932-49ce-aece-f9498937ecad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13943829-jl86x\" (UID: \"7238eabc-b932-49ce-aece-f9498937ecad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13943829-jl86x" Nov 03 18:19:42.116916 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:42.116851 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm9mp\" (UniqueName: \"kubernetes.io/projected/7238eabc-b932-49ce-aece-f9498937ecad-kube-api-access-mm9mp\") pod \"maas-api-revoke-sub-13943829-jl86x\" (UID: \"7238eabc-b932-49ce-aece-f9498937ecad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13943829-jl86x" Nov 03 18:19:42.116962 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:42.116927 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7238eabc-b932-49ce-aece-f9498937ecad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13943829-jl86x\" (UID: \"7238eabc-b932-49ce-aece-f9498937ecad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13943829-jl86x" Nov 03 18:19:42.117588 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:42.117569 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7238eabc-b932-49ce-aece-f9498937ecad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13943829-jl86x\" (UID: \"7238eabc-b932-49ce-aece-f9498937ecad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13943829-jl86x" Nov 03 18:19:42.119468 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:42.119414 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7238eabc-b932-49ce-aece-f9498937ecad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13943829-jl86x\" (UID: \"7238eabc-b932-49ce-aece-f9498937ecad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13943829-jl86x" Nov 03 18:19:42.174210 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:42.174176 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm9mp\" (UniqueName: \"kubernetes.io/projected/7238eabc-b932-49ce-aece-f9498937ecad-kube-api-access-mm9mp\") pod \"maas-api-revoke-sub-13943829-jl86x\" (UID: \"7238eabc-b932-49ce-aece-f9498937ecad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13943829-jl86x" Nov 03 18:19:42.185865 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:42.185841 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13943829-jl86x" Nov 03 18:19:42.553415 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:42.553322 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13943829-jl86x"] Nov 03 18:19:42.561190 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:19:42.561155 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7238eabc_b932_49ce_aece_f9498937ecad.slice/crio-c34aca461a28e6768e085488d9bcfdc868157e24fb46d13b779d2d91e47e6813 WatchSource:0}: Error finding container c34aca461a28e6768e085488d9bcfdc868157e24fb46d13b779d2d91e47e6813: Status 404 returned error can't find the container with id c34aca461a28e6768e085488d9bcfdc868157e24fb46d13b779d2d91e47e6813 Nov 03 18:19:43.247344 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:43.247264 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13943829-jl86x" event={"ID":"7238eabc-b932-49ce-aece-f9498937ecad","Type":"ContainerStarted","Data":"6ea7abd9fde926e5d8a2ccdf5a788b9d1f827fdf7983877e5a129f02ac7b457f"} Nov 03 18:19:43.247344 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:43.247315 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13943829-jl86x" event={"ID":"7238eabc-b932-49ce-aece-f9498937ecad","Type":"ContainerStarted","Data":"c34aca461a28e6768e085488d9bcfdc868157e24fb46d13b779d2d91e47e6813"} Nov 03 18:19:43.356682 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:43.356637 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13943829-jl86x" podStartSLOduration=2.356623378 podStartE2EDuration="2.356623378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:19:43.354447606 +0000 UTC m=+2020.488076609" watchObservedRunningTime="2026-11-03 18:19:43.356623378 +0000 UTC m=+2020.490252388" Nov 03 18:19:44.253926 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:44.253889 5805 generic.go:356] "Generic (PLEG): container finished" podID="7238eabc-b932-49ce-aece-f9498937ecad" containerID="6ea7abd9fde926e5d8a2ccdf5a788b9d1f827fdf7983877e5a129f02ac7b457f" exitCode=0 Nov 03 18:19:44.254329 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:44.253934 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13943829-jl86x" event={"ID":"7238eabc-b932-49ce-aece-f9498937ecad","Type":"ContainerDied","Data":"6ea7abd9fde926e5d8a2ccdf5a788b9d1f827fdf7983877e5a129f02ac7b457f"} Nov 03 18:19:45.396690 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:45.396667 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13943829-jl86x" Nov 03 18:19:45.553039 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:45.552964 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7238eabc-b932-49ce-aece-f9498937ecad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7238eabc-b932-49ce-aece-f9498937ecad-maas-api-openshift-ca\") pod \"7238eabc-b932-49ce-aece-f9498937ecad\" (UID: \"7238eabc-b932-49ce-aece-f9498937ecad\") " Nov 03 18:19:45.553208 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:45.553078 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7238eabc-b932-49ce-aece-f9498937ecad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7238eabc-b932-49ce-aece-f9498937ecad-maas-api-xks-ca\") pod \"7238eabc-b932-49ce-aece-f9498937ecad\" (UID: \"7238eabc-b932-49ce-aece-f9498937ecad\") " Nov 03 18:19:45.553208 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:45.553146 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7238eabc-b932-49ce-aece-f9498937ecad-kube-api-access-mm9mp\" (UniqueName: \"kubernetes.io/projected/7238eabc-b932-49ce-aece-f9498937ecad-kube-api-access-mm9mp\") pod \"7238eabc-b932-49ce-aece-f9498937ecad\" (UID: \"7238eabc-b932-49ce-aece-f9498937ecad\") " Nov 03 18:19:45.553382 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:45.553357 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7238eabc-b932-49ce-aece-f9498937ecad-maas-api-openshift-ca" pod "7238eabc-b932-49ce-aece-f9498937ecad" (UID: "7238eabc-b932-49ce-aece-f9498937ecad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:19:45.555184 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:45.555156 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7238eabc-b932-49ce-aece-f9498937ecad-maas-api-xks-ca" pod "7238eabc-b932-49ce-aece-f9498937ecad" (UID: "7238eabc-b932-49ce-aece-f9498937ecad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:19:45.555184 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:45.555172 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7238eabc-b932-49ce-aece-f9498937ecad-kube-api-access-mm9mp" pod "7238eabc-b932-49ce-aece-f9498937ecad" (UID: "7238eabc-b932-49ce-aece-f9498937ecad"). InnerVolumeSpecName "kube-api-access-mm9mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:19:45.654157 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:45.654125 5805 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7238eabc-b932-49ce-aece-f9498937ecad-maas-api-xks-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:19:45.654157 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:45.654151 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm9mp\" (UniqueName: \"kubernetes.io/projected/7238eabc-b932-49ce-aece-f9498937ecad-kube-api-access-mm9mp\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:19:45.654157 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:45.654162 5805 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7238eabc-b932-49ce-aece-f9498937ecad-maas-api-openshift-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:19:46.265202 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.265170 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13943829-jl86x" Nov 03 18:19:46.265374 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.265206 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13943829-jl86x" event={"ID":"7238eabc-b932-49ce-aece-f9498937ecad","Type":"ContainerDied","Data":"c34aca461a28e6768e085488d9bcfdc868157e24fb46d13b779d2d91e47e6813"} Nov 03 18:19:46.265374 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.265254 5805 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c34aca461a28e6768e085488d9bcfdc868157e24fb46d13b779d2d91e47e6813" Nov 03 18:19:46.588745 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.588685 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" podUID="ec4a7156-46e6-4274-9794-e6d1f14aec3f" containerName="main" containerID="cri-o://d5ab982aff08f0268827f8857a4e6f4598750a2bc60ed39be8c8abe3650e0a5e" gracePeriod=45 Nov 03 18:19:46.616971 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.616922 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" podUID="e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" containerName="main" containerID="cri-o://4310621185f485d9d9e07507103cdab04be1e5c652c8b899fd6bcc423164f4e6" gracePeriod=45 Nov 03 18:19:46.863935 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.863914 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:19:46.886839 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.886818 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 18:19:46.967342 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.967318 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-tls-certs\" (UniqueName: \"kubernetes.io/secret/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-tls-certs\") pod \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\" (UID: \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\") " Nov 03 18:19:46.967494 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.967356 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec4a7156-46e6-4274-9794-e6d1f14aec3f-kube-api-access-8vxws\" (UniqueName: \"kubernetes.io/projected/ec4a7156-46e6-4274-9794-e6d1f14aec3f-kube-api-access-8vxws\") pod \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\" (UID: \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\") " Nov 03 18:19:46.967494 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.967377 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-tmp-dir\") pod \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\" (UID: \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\") " Nov 03 18:19:46.967623 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.967529 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec4a7156-46e6-4274-9794-e6d1f14aec3f-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec4a7156-46e6-4274-9794-e6d1f14aec3f-tls-certs\") pod \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\" (UID: \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\") " Nov 03 18:19:46.967623 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.967575 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-model-cache\") pod \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\" (UID: \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\") " Nov 03 18:19:46.967623 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.967614 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-dshm\") pod \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\" (UID: \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\") " Nov 03 18:19:46.967792 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.967652 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-tmp-dir" pod "e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" (UID: "e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:19:46.967792 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.967710 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-kube-api-access-s7gl9\" (UniqueName: \"kubernetes.io/projected/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-kube-api-access-s7gl9\") pod \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\" (UID: \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\") " Nov 03 18:19:46.967792 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.967740 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-tmp-dir\") pod \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\" (UID: \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\") " Nov 03 18:19:46.967961 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.967804 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-model-cache\") pod \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\" (UID: \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\") " Nov 03 18:19:46.967961 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.967848 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-dshm\") pod \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\" (UID: \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\") " Nov 03 18:19:46.967961 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.967849 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-model-cache" pod "e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" (UID: "e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:19:46.967961 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.967874 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-home\" (UniqueName: \"kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-home\") pod \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\" (UID: \"ec4a7156-46e6-4274-9794-e6d1f14aec3f\") " Nov 03 18:19:46.967961 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.967921 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-home\" (UniqueName: \"kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-home\") pod \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\" (UID: \"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2\") " Nov 03 18:19:46.968270 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.968079 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-model-cache" pod "ec4a7156-46e6-4274-9794-e6d1f14aec3f" (UID: "ec4a7156-46e6-4274-9794-e6d1f14aec3f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:19:46.968270 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.968263 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-tmp-dir" pod "ec4a7156-46e6-4274-9794-e6d1f14aec3f" (UID: "ec4a7156-46e6-4274-9794-e6d1f14aec3f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:19:46.968409 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.968378 5805 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-tmp-dir\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:19:46.968479 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.968415 5805 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-model-cache\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:19:46.968479 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.968460 5805 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-tmp-dir\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:19:46.968479 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.968475 5805 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-model-cache\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:19:46.968656 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.968487 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-home" pod "ec4a7156-46e6-4274-9794-e6d1f14aec3f" (UID: "ec4a7156-46e6-4274-9794-e6d1f14aec3f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:19:46.968718 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.968658 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-home" pod "e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" (UID: "e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:19:46.969936 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.969911 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-tls-certs" pod "e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" (UID: "e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:19:46.970037 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.969909 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec4a7156-46e6-4274-9794-e6d1f14aec3f-tls-certs" pod "ec4a7156-46e6-4274-9794-e6d1f14aec3f" (UID: "ec4a7156-46e6-4274-9794-e6d1f14aec3f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:19:46.970037 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.969979 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4a7156-46e6-4274-9794-e6d1f14aec3f-kube-api-access-8vxws" pod "ec4a7156-46e6-4274-9794-e6d1f14aec3f" (UID: "ec4a7156-46e6-4274-9794-e6d1f14aec3f"). InnerVolumeSpecName "kube-api-access-8vxws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:19:46.970117 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.970063 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-dshm" pod "e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" (UID: "e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:19:46.970270 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.970253 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-dshm" pod "ec4a7156-46e6-4274-9794-e6d1f14aec3f" (UID: "ec4a7156-46e6-4274-9794-e6d1f14aec3f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:19:46.971309 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:46.971290 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-kube-api-access-s7gl9" pod "e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" (UID: "e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2"). InnerVolumeSpecName "kube-api-access-s7gl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:19:47.069597 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.069564 5805 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec4a7156-46e6-4274-9794-e6d1f14aec3f-tls-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:19:47.069597 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.069591 5805 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-dshm\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:19:47.069788 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.069605 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7gl9\" (UniqueName: \"kubernetes.io/projected/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-kube-api-access-s7gl9\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:19:47.069788 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.069618 5805 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-dshm\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:19:47.069788 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.069630 5805 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec4a7156-46e6-4274-9794-e6d1f14aec3f-home\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:19:47.069788 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.069641 5805 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-home\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:19:47.069788 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.069652 5805 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2-tls-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:19:47.069788 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.069664 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vxws\" (UniqueName: \"kubernetes.io/projected/ec4a7156-46e6-4274-9794-e6d1f14aec3f-kube-api-access-8vxws\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:19:47.272728 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.272640 5805 generic.go:356] "Generic (PLEG): container finished" podID="e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" containerID="4310621185f485d9d9e07507103cdab04be1e5c652c8b899fd6bcc423164f4e6" exitCode=0 Nov 03 18:19:47.272728 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.272688 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" Nov 03 18:19:47.272946 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.272729 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" event={"ID":"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2","Type":"ContainerDied","Data":"4310621185f485d9d9e07507103cdab04be1e5c652c8b899fd6bcc423164f4e6"} Nov 03 18:19:47.272946 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.272782 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l" event={"ID":"e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2","Type":"ContainerDied","Data":"5f85a7ad6ff9857bb5c5dffa2998d9668ece862cafab7c75add2aeb41b1c2b04"} Nov 03 18:19:47.272946 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.272806 5805 scope.go:122] "RemoveContainer" containerID="4310621185f485d9d9e07507103cdab04be1e5c652c8b899fd6bcc423164f4e6" Nov 03 18:19:47.274998 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.274828 5805 generic.go:356] "Generic (PLEG): container finished" podID="ec4a7156-46e6-4274-9794-e6d1f14aec3f" containerID="d5ab982aff08f0268827f8857a4e6f4598750a2bc60ed39be8c8abe3650e0a5e" exitCode=0 Nov 03 18:19:47.274998 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.274901 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" event={"ID":"ec4a7156-46e6-4274-9794-e6d1f14aec3f","Type":"ContainerDied","Data":"d5ab982aff08f0268827f8857a4e6f4598750a2bc60ed39be8c8abe3650e0a5e"} Nov 03 18:19:47.274998 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.274957 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" event={"ID":"ec4a7156-46e6-4274-9794-e6d1f14aec3f","Type":"ContainerDied","Data":"c79540f06d5cd02ce00e25e3b12724edd02d1474381b249d720ecf6493be752d"} Nov 03 18:19:47.275241 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.275133 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf" Nov 03 18:19:47.283439 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.283420 5805 scope.go:122] "RemoveContainer" containerID="4310621185f485d9d9e07507103cdab04be1e5c652c8b899fd6bcc423164f4e6" Nov 03 18:19:47.283767 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:19:47.283739 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4310621185f485d9d9e07507103cdab04be1e5c652c8b899fd6bcc423164f4e6\": container with ID starting with 4310621185f485d9d9e07507103cdab04be1e5c652c8b899fd6bcc423164f4e6 not found: ID does not exist" containerID="4310621185f485d9d9e07507103cdab04be1e5c652c8b899fd6bcc423164f4e6" Nov 03 18:19:47.283864 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.283773 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4310621185f485d9d9e07507103cdab04be1e5c652c8b899fd6bcc423164f4e6"} err="failed to get container status \"4310621185f485d9d9e07507103cdab04be1e5c652c8b899fd6bcc423164f4e6\": rpc error: code = NotFound desc = could not find container \"4310621185f485d9d9e07507103cdab04be1e5c652c8b899fd6bcc423164f4e6\": container with ID starting with 4310621185f485d9d9e07507103cdab04be1e5c652c8b899fd6bcc423164f4e6 not found: ID does not exist" Nov 03 18:19:47.283864 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.283791 5805 scope.go:122] "RemoveContainer" containerID="d5ab982aff08f0268827f8857a4e6f4598750a2bc60ed39be8c8abe3650e0a5e" Nov 03 18:19:47.297965 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.297937 5805 scope.go:122] "RemoveContainer" containerID="d5ab982aff08f0268827f8857a4e6f4598750a2bc60ed39be8c8abe3650e0a5e" Nov 03 18:19:47.298248 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:19:47.298223 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5ab982aff08f0268827f8857a4e6f4598750a2bc60ed39be8c8abe3650e0a5e\": container with ID starting with d5ab982aff08f0268827f8857a4e6f4598750a2bc60ed39be8c8abe3650e0a5e not found: ID does not exist" containerID="d5ab982aff08f0268827f8857a4e6f4598750a2bc60ed39be8c8abe3650e0a5e" Nov 03 18:19:47.298356 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.298256 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5ab982aff08f0268827f8857a4e6f4598750a2bc60ed39be8c8abe3650e0a5e"} err="failed to get container status \"d5ab982aff08f0268827f8857a4e6f4598750a2bc60ed39be8c8abe3650e0a5e\": rpc error: code = NotFound desc = could not find container \"d5ab982aff08f0268827f8857a4e6f4598750a2bc60ed39be8c8abe3650e0a5e\": container with ID starting with d5ab982aff08f0268827f8857a4e6f4598750a2bc60ed39be8c8abe3650e0a5e not found: ID does not exist" Nov 03 18:19:47.426440 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.426350 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l"] Nov 03 18:19:47.432143 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.432117 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-ef9633/simulator-w1-ef9633-kserve-5867bfbd86-rjh4l"] Nov 03 18:19:47.533097 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.533030 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf"] Nov 03 18:19:47.538073 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:47.538047 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-ef9633/distinct-2-w1-ef9633-kserve-66c74b94df-2f4sf"] Nov 03 18:19:48.981847 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:48.981811 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" path="/var/lib/kubelet/pods/e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2/volumes" Nov 03 18:19:48.982304 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:48.982216 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec4a7156-46e6-4274-9794-e6d1f14aec3f" path="/var/lib/kubelet/pods/ec4a7156-46e6-4274-9794-e6d1f14aec3f/volumes" Nov 03 18:19:56.235221 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:19:56.235181 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:19:57.257716 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:57.257671 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:19:58.708671 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:58.708634 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:19:58.983009 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:58.982893 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:19:59.755358 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:19:59.755321 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:20:06.673631 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:06.673569 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:20:06.881316 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:06.881283 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:20:06.964806 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:06.964717 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn"] Nov 03 18:20:06.965027 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:06.964995 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" podUID="f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4" containerName="istio-proxy" containerID="cri-o://97755c9f8d76b8f059d8dc6f19b2800e415153b4e33f89ae39dfddf0cd9202f1" gracePeriod=30 Nov 03 18:20:07.083271 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:07.083238 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:20:08.639781 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:08.639731 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:20:08.840352 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:08.840306 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:20:09.485333 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:09.485293 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:20:12.210626 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.210599 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:20:12.334968 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.334930 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c8b7c4cfc-4lkmm"] Nov 03 18:20:12.335157 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.335126 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5c8b7c4cfc-4lkmm" podUID="60dc8570-85f6-4133-9567-5e99139b44e7" containerName="manager" containerID="cri-o://341194b77799fcd79668dcdf981c3d8d0e8c27f3a5d5f6cc8a0086f3f2a00fc9" gracePeriod=30 Nov 03 18:20:12.337848 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.337804 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-kube-api-access-6dc4m\" (UniqueName: \"kubernetes.io/projected/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-kube-api-access-6dc4m\") pod \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " Nov 03 18:20:12.338010 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.337871 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-data\") pod \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " Nov 03 18:20:12.338010 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.337901 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-token\" (UniqueName: \"kubernetes.io/projected/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-token\") pod \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " Nov 03 18:20:12.338010 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.337971 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-credential-socket\") pod \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " Nov 03 18:20:12.338010 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.338003 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-podinfo\") pod \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " Nov 03 18:20:12.338248 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.338057 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istiod-ca-cert\") pod \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " Nov 03 18:20:12.338248 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.338106 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-envoy\") pod \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " Nov 03 18:20:12.338248 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.338154 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-workload-socket\") pod \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " Nov 03 18:20:12.338248 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.338189 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-workload-certs\") pod \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\" (UID: \"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4\") " Nov 03 18:20:12.338689 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.338628 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-credential-socket" pod "f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4" (UID: "f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:12.338809 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.338700 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-workload-certs" pod "f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4" (UID: "f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:12.338878 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.338835 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-data" pod "f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4" (UID: "f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:12.338937 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.338910 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-workload-socket" pod "f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4" (UID: "f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:12.339001 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.338940 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istiod-ca-cert" pod "f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4" (UID: "f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:20:12.340868 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.340846 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-podinfo" pod "f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4" (UID: "f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:20:12.340965 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.340882 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-kube-api-access-6dc4m" pod "f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4" (UID: "f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4"). InnerVolumeSpecName "kube-api-access-6dc4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:20:12.342867 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.342846 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-envoy" pod "f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4" (UID: "f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:12.343057 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.343036 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-token" pod "f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4" (UID: "f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:20:12.413387 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.413342 5805 generic.go:356] "Generic (PLEG): container finished" podID="f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4" containerID="97755c9f8d76b8f059d8dc6f19b2800e415153b4e33f89ae39dfddf0cd9202f1" exitCode=0 Nov 03 18:20:12.413593 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.413411 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" Nov 03 18:20:12.413593 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.413484 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" event={"ID":"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4","Type":"ContainerDied","Data":"97755c9f8d76b8f059d8dc6f19b2800e415153b4e33f89ae39dfddf0cd9202f1"} Nov 03 18:20:12.413593 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.413529 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn" event={"ID":"f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4","Type":"ContainerDied","Data":"59a06c829e54835d8112d13cfc7898cea0160b2024f0b718913ce1fedba2eeb9"} Nov 03 18:20:12.413593 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.413550 5805 scope.go:122] "RemoveContainer" containerID="97755c9f8d76b8f059d8dc6f19b2800e415153b4e33f89ae39dfddf0cd9202f1" Nov 03 18:20:12.423712 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.423693 5805 scope.go:122] "RemoveContainer" containerID="97755c9f8d76b8f059d8dc6f19b2800e415153b4e33f89ae39dfddf0cd9202f1" Nov 03 18:20:12.423993 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:20:12.423972 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97755c9f8d76b8f059d8dc6f19b2800e415153b4e33f89ae39dfddf0cd9202f1\": container with ID starting with 97755c9f8d76b8f059d8dc6f19b2800e415153b4e33f89ae39dfddf0cd9202f1 not found: ID does not exist" containerID="97755c9f8d76b8f059d8dc6f19b2800e415153b4e33f89ae39dfddf0cd9202f1" Nov 03 18:20:12.424046 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.424009 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97755c9f8d76b8f059d8dc6f19b2800e415153b4e33f89ae39dfddf0cd9202f1"} err="failed to get container status \"97755c9f8d76b8f059d8dc6f19b2800e415153b4e33f89ae39dfddf0cd9202f1\": rpc error: code = NotFound desc = could not find container \"97755c9f8d76b8f059d8dc6f19b2800e415153b4e33f89ae39dfddf0cd9202f1\": container with ID starting with 97755c9f8d76b8f059d8dc6f19b2800e415153b4e33f89ae39dfddf0cd9202f1 not found: ID does not exist" Nov 03 18:20:12.440062 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.440033 5805 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-credential-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:20:12.440168 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.440066 5805 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-podinfo\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:20:12.440168 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.440082 5805 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istiod-ca-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:20:12.440168 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.440096 5805 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-envoy\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:20:12.440168 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.440109 5805 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-workload-socket\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:20:12.440168 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.440123 5805 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-workload-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:20:12.440168 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.440137 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dc4m\" (UniqueName: \"kubernetes.io/projected/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-kube-api-access-6dc4m\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:20:12.440168 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.440152 5805 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-data\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:20:12.440168 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.440165 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:20:12.572318 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.572290 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c8b7c4cfc-4lkmm" Nov 03 18:20:12.585979 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.585923 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn"] Nov 03 18:20:12.590415 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.590368 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-ef9633-openshift-default-98f444cc5-jg4qn"] Nov 03 18:20:12.743348 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.743315 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60dc8570-85f6-4133-9567-5e99139b44e7-webhook-cert\" (UniqueName: \"kubernetes.io/secret/60dc8570-85f6-4133-9567-5e99139b44e7-webhook-cert\") pod \"60dc8570-85f6-4133-9567-5e99139b44e7\" (UID: \"60dc8570-85f6-4133-9567-5e99139b44e7\") " Nov 03 18:20:12.743600 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.743369 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60dc8570-85f6-4133-9567-5e99139b44e7-kube-api-access-fvg4z\" (UniqueName: \"kubernetes.io/projected/60dc8570-85f6-4133-9567-5e99139b44e7-kube-api-access-fvg4z\") pod \"60dc8570-85f6-4133-9567-5e99139b44e7\" (UID: \"60dc8570-85f6-4133-9567-5e99139b44e7\") " Nov 03 18:20:12.743600 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.743465 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60dc8570-85f6-4133-9567-5e99139b44e7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/60dc8570-85f6-4133-9567-5e99139b44e7-metrics-certs\") pod \"60dc8570-85f6-4133-9567-5e99139b44e7\" (UID: \"60dc8570-85f6-4133-9567-5e99139b44e7\") " Nov 03 18:20:12.745409 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.745362 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60dc8570-85f6-4133-9567-5e99139b44e7-webhook-cert" pod "60dc8570-85f6-4133-9567-5e99139b44e7" (UID: "60dc8570-85f6-4133-9567-5e99139b44e7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:20:12.745916 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.745892 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60dc8570-85f6-4133-9567-5e99139b44e7-metrics-certs" pod "60dc8570-85f6-4133-9567-5e99139b44e7" (UID: "60dc8570-85f6-4133-9567-5e99139b44e7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:20:12.745916 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.745911 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60dc8570-85f6-4133-9567-5e99139b44e7-kube-api-access-fvg4z" pod "60dc8570-85f6-4133-9567-5e99139b44e7" (UID: "60dc8570-85f6-4133-9567-5e99139b44e7"). InnerVolumeSpecName "kube-api-access-fvg4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:20:12.844693 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.844617 5805 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60dc8570-85f6-4133-9567-5e99139b44e7-metrics-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:20:12.844693 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.844648 5805 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60dc8570-85f6-4133-9567-5e99139b44e7-webhook-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:20:12.844693 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.844659 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvg4z\" (UniqueName: \"kubernetes.io/projected/60dc8570-85f6-4133-9567-5e99139b44e7-kube-api-access-fvg4z\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:20:12.975297 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:12.975256 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4" path="/var/lib/kubelet/pods/f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4/volumes" Nov 03 18:20:13.421031 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:13.420998 5805 generic.go:356] "Generic (PLEG): container finished" podID="60dc8570-85f6-4133-9567-5e99139b44e7" containerID="341194b77799fcd79668dcdf981c3d8d0e8c27f3a5d5f6cc8a0086f3f2a00fc9" exitCode=0 Nov 03 18:20:13.421484 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:13.421048 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c8b7c4cfc-4lkmm" event={"ID":"60dc8570-85f6-4133-9567-5e99139b44e7","Type":"ContainerDied","Data":"341194b77799fcd79668dcdf981c3d8d0e8c27f3a5d5f6cc8a0086f3f2a00fc9"} Nov 03 18:20:13.421484 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:13.421084 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c8b7c4cfc-4lkmm" event={"ID":"60dc8570-85f6-4133-9567-5e99139b44e7","Type":"ContainerDied","Data":"cf299eee582e7c015aed7f8ce0f5eb6d358bdc718fc34bc1616857c8e087c58e"} Nov 03 18:20:13.421484 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:13.421106 5805 scope.go:122] "RemoveContainer" containerID="341194b77799fcd79668dcdf981c3d8d0e8c27f3a5d5f6cc8a0086f3f2a00fc9" Nov 03 18:20:13.421484 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:13.421053 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c8b7c4cfc-4lkmm" Nov 03 18:20:13.430371 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:13.430345 5805 scope.go:122] "RemoveContainer" containerID="341194b77799fcd79668dcdf981c3d8d0e8c27f3a5d5f6cc8a0086f3f2a00fc9" Nov 03 18:20:13.430780 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:20:13.430758 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"341194b77799fcd79668dcdf981c3d8d0e8c27f3a5d5f6cc8a0086f3f2a00fc9\": container with ID starting with 341194b77799fcd79668dcdf981c3d8d0e8c27f3a5d5f6cc8a0086f3f2a00fc9 not found: ID does not exist" containerID="341194b77799fcd79668dcdf981c3d8d0e8c27f3a5d5f6cc8a0086f3f2a00fc9" Nov 03 18:20:13.430890 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:13.430785 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341194b77799fcd79668dcdf981c3d8d0e8c27f3a5d5f6cc8a0086f3f2a00fc9"} err="failed to get container status \"341194b77799fcd79668dcdf981c3d8d0e8c27f3a5d5f6cc8a0086f3f2a00fc9\": rpc error: code = NotFound desc = could not find container \"341194b77799fcd79668dcdf981c3d8d0e8c27f3a5d5f6cc8a0086f3f2a00fc9\": container with ID starting with 341194b77799fcd79668dcdf981c3d8d0e8c27f3a5d5f6cc8a0086f3f2a00fc9 not found: ID does not exist" Nov 03 18:20:13.574171 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:13.574140 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c8b7c4cfc-4lkmm"] Nov 03 18:20:13.620011 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:13.619983 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5c8b7c4cfc-4lkmm"] Nov 03 18:20:14.975918 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:14.975875 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60dc8570-85f6-4133-9567-5e99139b44e7" path="/var/lib/kubelet/pods/60dc8570-85f6-4133-9567-5e99139b44e7/volumes" Nov 03 18:20:15.541423 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:15.541000 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:20:16.050911 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:16.050859 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9525b615-wtwgt"] Nov 03 18:20:16.095759 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:16.095720 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9525b615-wtwgt"] Nov 03 18:20:16.987422 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:16.986525 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00a224da-b74e-4186-b7ef-030cc5fc1c92" path="/var/lib/kubelet/pods/00a224da-b74e-4186-b7ef-030cc5fc1c92/volumes" Nov 03 18:20:17.053249 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:17.053212 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:20:17.057808 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:17.057785 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-712e5ae1-s7d65"] Nov 03 18:20:17.093149 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:17.093099 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-712e5ae1-s7d65"] Nov 03 18:20:18.974848 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:18.974815 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce5323ef-e7f2-4921-bed5-9e71e2aa835d" path="/var/lib/kubelet/pods/ce5323ef-e7f2-4921-bed5-9e71e2aa835d/volumes" Nov 03 18:20:26.235408 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:20:26.235358 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:20:45.523323 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:45.523247 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:20:45.727548 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:45.727516 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:20:51.953224 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:51.953186 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:20:53.930988 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:20:53.930956 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:20:56.235044 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:20:56.235004 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:21:01.920834 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:21:01.920794 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:21:02.117375 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:21:02.117343 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:21:03.311985 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:21:03.311957 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 18:21:03.314295 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:21:03.314275 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 18:21:17.313899 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:21:17.313863 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:21:17.544862 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:21:17.544797 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:21:26.234258 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:21:26.234221 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:21:26.272103 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:21:26.272072 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:21:26.464061 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:21:26.464023 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:21:55.981017 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:21:55.980986 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:21:56.183744 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:21:56.183714 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:21:56.234157 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:21:56.234076 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:22:03.581779 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:22:03.581741 5805 scope.go:122] "RemoveContainer" containerID="85e0a826c32c9d2125af7225ba0692609093410d5bad6ac7076952427655c743" Nov 03 18:22:03.595014 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:22:03.594996 5805 scope.go:122] "RemoveContainer" containerID="0692b73d3ecb0867617afc43dd9aebcfcae94523be347fe650282748e9cbe1a7" Nov 03 18:22:03.602427 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:22:03.602405 5805 scope.go:122] "RemoveContainer" containerID="eec8d45ab6b3c2785110373653eaa9e371f7cfd603eac9722e703127e47f2f00" Nov 03 18:22:06.271560 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:22:06.271517 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6stkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:22:06.272194 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:22:06.272142 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" podUID="c6669c77-7d2c-48ba-829f-85ce6e6f9f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:22:13.752590 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:22:13.752510 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:22:13.948309 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:22:13.948225 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:22:17.239884 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:22:17.239825 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:22:17.604764 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:22:17.604726 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:22:26.234015 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:22:26.233976 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:22:29.676361 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:22:29.676313 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:22:50.801757 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:22:50.801715 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:22:50.995860 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:22:50.995822 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:22:56.234026 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:22:56.233991 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:23:04.361964 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:04.361929 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf"] Nov 03 18:23:04.362472 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:04.362105 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" podUID="e8856532-061a-4f17-b49c-5e92228b7c02" containerName="discovery" containerID="cri-o://be1e8177cb462673180709120879445ee22e3b07a232a5a475448a157a360f6e" gracePeriod=30 Nov 03 18:23:04.614299 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:04.614219 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:23:04.643190 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:04.643158 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8856532-061a-4f17-b49c-5e92228b7c02-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e8856532-061a-4f17-b49c-5e92228b7c02-istio-kubeconfig\") pod \"e8856532-061a-4f17-b49c-5e92228b7c02\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " Nov 03 18:23:04.643357 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:04.643216 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8856532-061a-4f17-b49c-5e92228b7c02-cacerts\" (UniqueName: \"kubernetes.io/secret/e8856532-061a-4f17-b49c-5e92228b7c02-cacerts\") pod \"e8856532-061a-4f17-b49c-5e92228b7c02\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " Nov 03 18:23:04.643357 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:04.643255 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8856532-061a-4f17-b49c-5e92228b7c02-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e8856532-061a-4f17-b49c-5e92228b7c02-istio-csr-dns-cert\") pod \"e8856532-061a-4f17-b49c-5e92228b7c02\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " Nov 03 18:23:04.643357 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:04.643300 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8856532-061a-4f17-b49c-5e92228b7c02-kube-api-access-g7dq6\" (UniqueName: \"kubernetes.io/projected/e8856532-061a-4f17-b49c-5e92228b7c02-kube-api-access-g7dq6\") pod \"e8856532-061a-4f17-b49c-5e92228b7c02\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " Nov 03 18:23:04.643357 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:04.643352 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8856532-061a-4f17-b49c-5e92228b7c02-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8856532-061a-4f17-b49c-5e92228b7c02-local-certs\") pod \"e8856532-061a-4f17-b49c-5e92228b7c02\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " Nov 03 18:23:04.643620 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:04.643435 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8856532-061a-4f17-b49c-5e92228b7c02-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e8856532-061a-4f17-b49c-5e92228b7c02-istio-csr-ca-configmap\") pod \"e8856532-061a-4f17-b49c-5e92228b7c02\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " Nov 03 18:23:04.643620 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:04.643555 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8856532-061a-4f17-b49c-5e92228b7c02-istio-token\" (UniqueName: \"kubernetes.io/projected/e8856532-061a-4f17-b49c-5e92228b7c02-istio-token\") pod \"e8856532-061a-4f17-b49c-5e92228b7c02\" (UID: \"e8856532-061a-4f17-b49c-5e92228b7c02\") " Nov 03 18:23:04.643967 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:04.643930 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8856532-061a-4f17-b49c-5e92228b7c02-istio-csr-ca-configmap" pod "e8856532-061a-4f17-b49c-5e92228b7c02" (UID: "e8856532-061a-4f17-b49c-5e92228b7c02"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:23:04.645856 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:04.645828 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8856532-061a-4f17-b49c-5e92228b7c02-istio-csr-dns-cert" pod "e8856532-061a-4f17-b49c-5e92228b7c02" (UID: "e8856532-061a-4f17-b49c-5e92228b7c02"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:23:04.645997 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:04.645850 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8856532-061a-4f17-b49c-5e92228b7c02-istio-kubeconfig" pod "e8856532-061a-4f17-b49c-5e92228b7c02" (UID: "e8856532-061a-4f17-b49c-5e92228b7c02"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:23:04.646167 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:04.646137 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8856532-061a-4f17-b49c-5e92228b7c02-cacerts" pod "e8856532-061a-4f17-b49c-5e92228b7c02" (UID: "e8856532-061a-4f17-b49c-5e92228b7c02"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:23:04.646167 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:04.646159 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8856532-061a-4f17-b49c-5e92228b7c02-kube-api-access-g7dq6" pod "e8856532-061a-4f17-b49c-5e92228b7c02" (UID: "e8856532-061a-4f17-b49c-5e92228b7c02"). InnerVolumeSpecName "kube-api-access-g7dq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:23:04.646310 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:04.646172 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8856532-061a-4f17-b49c-5e92228b7c02-istio-token" pod "e8856532-061a-4f17-b49c-5e92228b7c02" (UID: "e8856532-061a-4f17-b49c-5e92228b7c02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:23:04.646310 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:04.646219 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8856532-061a-4f17-b49c-5e92228b7c02-local-certs" pod "e8856532-061a-4f17-b49c-5e92228b7c02" (UID: "e8856532-061a-4f17-b49c-5e92228b7c02"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:23:04.745300 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:04.745274 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7dq6\" (UniqueName: \"kubernetes.io/projected/e8856532-061a-4f17-b49c-5e92228b7c02-kube-api-access-g7dq6\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:23:04.745300 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:04.745299 5805 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8856532-061a-4f17-b49c-5e92228b7c02-local-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:23:04.745507 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:04.745311 5805 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e8856532-061a-4f17-b49c-5e92228b7c02-istio-csr-ca-configmap\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:23:04.745507 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:04.745319 5805 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8856532-061a-4f17-b49c-5e92228b7c02-istio-token\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:23:04.745507 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:04.745328 5805 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e8856532-061a-4f17-b49c-5e92228b7c02-istio-kubeconfig\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:23:04.745507 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:04.745335 5805 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e8856532-061a-4f17-b49c-5e92228b7c02-cacerts\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:23:04.745507 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:04.745344 5805 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e8856532-061a-4f17-b49c-5e92228b7c02-istio-csr-dns-cert\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:23:05.377693 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:05.377662 5805 generic.go:356] "Generic (PLEG): container finished" podID="e8856532-061a-4f17-b49c-5e92228b7c02" containerID="be1e8177cb462673180709120879445ee22e3b07a232a5a475448a157a360f6e" exitCode=0 Nov 03 18:23:05.378084 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:05.377713 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" Nov 03 18:23:05.378084 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:05.377741 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" event={"ID":"e8856532-061a-4f17-b49c-5e92228b7c02","Type":"ContainerDied","Data":"be1e8177cb462673180709120879445ee22e3b07a232a5a475448a157a360f6e"} Nov 03 18:23:05.378084 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:05.377799 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf" event={"ID":"e8856532-061a-4f17-b49c-5e92228b7c02","Type":"ContainerDied","Data":"4b117d02c97d727aa612bd3c9af2ce4d3c9b9f72a3cc7fa48801f4c23c821c41"} Nov 03 18:23:05.378084 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:05.377825 5805 scope.go:122] "RemoveContainer" containerID="be1e8177cb462673180709120879445ee22e3b07a232a5a475448a157a360f6e" Nov 03 18:23:05.389050 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:05.389032 5805 scope.go:122] "RemoveContainer" containerID="be1e8177cb462673180709120879445ee22e3b07a232a5a475448a157a360f6e" Nov 03 18:23:05.389388 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:23:05.389355 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be1e8177cb462673180709120879445ee22e3b07a232a5a475448a157a360f6e\": container with ID starting with be1e8177cb462673180709120879445ee22e3b07a232a5a475448a157a360f6e not found: ID does not exist" containerID="be1e8177cb462673180709120879445ee22e3b07a232a5a475448a157a360f6e" Nov 03 18:23:05.389486 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:05.389417 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be1e8177cb462673180709120879445ee22e3b07a232a5a475448a157a360f6e"} err="failed to get container status \"be1e8177cb462673180709120879445ee22e3b07a232a5a475448a157a360f6e\": rpc error: code = NotFound desc = could not find container \"be1e8177cb462673180709120879445ee22e3b07a232a5a475448a157a360f6e\": container with ID starting with be1e8177cb462673180709120879445ee22e3b07a232a5a475448a157a360f6e not found: ID does not exist" Nov 03 18:23:05.578074 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:05.578042 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf"] Nov 03 18:23:05.624626 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:05.624593 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vhkvf"] Nov 03 18:23:06.975217 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:06.975183 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8856532-061a-4f17-b49c-5e92228b7c02" path="/var/lib/kubelet/pods/e8856532-061a-4f17-b49c-5e92228b7c02/volumes" Nov 03 18:23:17.975364 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:23:17.975327 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:23:26.234592 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:23:26.234551 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:23:56.234516 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:23:56.234440 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:24:18.293309 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:24:18.293265 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:24:18.489952 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:24:18.489921 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:24:26.234864 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:24:26.234824 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:24:46.048915 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:24:46.048879 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13943829-jl86x"] Nov 03 18:24:46.095375 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:24:46.095343 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13943829-jl86x"] Nov 03 18:24:46.974649 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:24:46.974619 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7238eabc-b932-49ce-aece-f9498937ecad" path="/var/lib/kubelet/pods/7238eabc-b932-49ce-aece-f9498937ecad/volumes" Nov 03 18:24:56.233990 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:24:56.233950 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:25:07.666552 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:25:07.666511 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jmn7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:25:07.675638 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:25:07.667081 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" podUID="3369b197-476f-4861-b6c3-ac9bfa5464d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:25:18.643211 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:25:18.643137 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:25:26.234963 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:25:26.234925 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:25:56.234455 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:25:56.234416 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:26:03.347760 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:26:03.347730 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 18:26:03.350425 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:26:03.350389 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 18:26:03.667762 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:26:03.667676 5805 scope.go:122] "RemoveContainer" containerID="6ea7abd9fde926e5d8a2ccdf5a788b9d1f827fdf7983877e5a129f02ac7b457f" Nov 03 18:26:18.978877 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:26:18.978830 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:26:26.234927 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:26:26.234886 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:26:29.120076 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:26:29.120022 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:26:56.233703 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:26:56.233623 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:27:07.207265 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:27:07.207232 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:27:07.823157 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:27:07.823113 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:27:08.025571 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:27:08.025537 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:27:11.837343 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:27:11.837310 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:27:12.042688 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:27:12.042648 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:27:12.342083 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:27:12.342050 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:27:15.130564 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:27:15.130532 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:27:15.337115 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:27:15.337082 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:27:18.366130 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:27:18.366088 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:27:18.577674 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:27:18.577641 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:27:18.776801 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:27:18.776716 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:27:19.329250 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:27:19.329220 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:27:23.306098 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:27:23.306053 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:27:26.242505 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:27:26.242460 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:27:28.213469 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:27:28.213429 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:27:33.016174 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:27:33.016135 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:27:36.704407 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:27:36.704359 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:27:37.288214 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:27:37.288177 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:27:37.575480 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:27:37.575449 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:27:38.217530 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:27:38.217496 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:27:40.009677 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:27:40.009629 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:27:43.014193 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:27:43.014161 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:27:47.921177 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:27:47.921140 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:27:52.826895 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:27:52.826861 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:27:56.233694 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:27:56.233656 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:27:57.703332 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:27:57.703297 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:28:02.509118 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:28:02.509086 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:28:07.410641 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:28:07.410605 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:28:12.227291 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:28:12.227218 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:28:17.108484 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:28:17.108445 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:28:19.674445 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:28:19.674406 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:28:21.914759 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:28:21.914719 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:28:26.233584 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:28:26.233546 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:28:26.812690 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:28:26.812653 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:28:31.615062 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:28:31.615024 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:28:36.506978 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:28:36.506943 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:28:36.748088 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:28:36.748053 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:28:41.524419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:28:41.524364 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:28:41.878596 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:28:41.878562 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:28:46.408119 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:28:46.408087 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:28:49.137157 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:28:49.137094 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:28:49.338793 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:28:49.338760 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:28:51.111248 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:28:51.111217 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:28:55.363150 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:28:55.363111 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:28:56.234353 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:28:56.234317 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:28:56.902677 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:28:56.902634 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:28:57.181461 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:28:57.181373 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:28:57.943755 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:28:57.943716 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:29:03.673978 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:29:03.673943 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:29:03.877737 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:29:03.877698 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:29:04.089374 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:29:04.089343 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:29:11.619525 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:29:11.619492 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:29:18.726580 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:29:18.726540 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:29:19.303731 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:29:19.303695 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:29:20.064747 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:29:20.064714 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:29:25.155031 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:29:25.154994 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:29:26.239217 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:29:26.239175 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:29:56.234276 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:29:56.234189 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:30:20.346938 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:30:20.346901 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:30:26.234780 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:30:26.234743 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:30:56.298179 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:30:56.298095 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:31:03.386275 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:31:03.386250 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 18:31:03.389234 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:31:03.389214 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 18:31:20.700874 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:31:20.700797 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:31:26.234277 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:31:26.234228 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:31:40.572439 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:31:40.572389 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:31:41.362191 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:31:41.362155 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:31:41.624913 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:31:41.624827 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:31:45.319205 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:31:45.319171 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:31:45.527486 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:31:45.527449 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:31:48.597568 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:31:48.597537 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:31:48.798644 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:31:48.798607 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:31:51.757826 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:31:51.757741 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:31:51.962886 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:31:51.962839 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:31:52.166404 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:31:52.166360 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:31:56.233142 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:31:56.233107 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:31:56.728834 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:31:56.728798 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:32:01.718108 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:32:01.718071 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:32:06.510335 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:32:06.510294 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:32:10.193520 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:32:10.193471 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:32:10.826438 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:32:10.826385 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:32:11.032095 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:32:11.032059 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:32:11.712115 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:32:11.712077 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:32:13.996971 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:32:13.996937 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:32:16.511854 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:32:16.511816 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:32:21.059532 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:32:21.059499 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:32:21.522507 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:32:21.522406 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:32:26.242181 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:32:26.242136 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:32:26.311932 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:32:26.311902 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:32:31.204583 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:32:31.204547 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:32:36.012240 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:32:36.012187 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:32:39.593742 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:32:39.593697 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:32:40.913504 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:32:40.913471 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:32:45.998287 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:32:45.998195 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:32:50.618808 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:32:50.618775 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:32:55.413841 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:32:55.413807 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:32:56.234675 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:32:56.234640 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:32:58.288974 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:32:58.288932 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:32:58.486637 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:32:58.486601 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:33:00.308969 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:33:00.308924 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:33:05.108958 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:33:05.108918 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:33:09.985573 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:33:09.985535 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:33:10.564800 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:33:10.564769 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:33:15.110907 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:33:15.110873 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:33:15.594010 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:33:15.593974 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:33:20.001706 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:33:20.001667 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:33:21.407233 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:33:21.407198 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:33:22.993006 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:33:22.992965 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:33:23.200152 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:33:23.200115 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:33:24.813536 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:33:24.813497 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:33:26.234460 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:33:26.234422 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:33:29.048957 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:33:29.048924 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:33:30.568797 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:33:30.568763 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:33:30.856301 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:33:30.856220 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:33:31.643267 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:33:31.643236 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:33:33.987110 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:33:33.987070 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:33:37.718067 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:33:37.718036 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:33:37.918771 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:33:37.918732 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:33:38.117620 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:33:38.117588 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:33:44.914744 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:33:44.914703 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:33:52.196999 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:33:52.196960 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:33:52.832111 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:33:52.832077 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:33:56.235190 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:33:56.235153 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:34:07.800454 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:34:07.800414 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:34:13.206056 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:34:13.205980 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:34:13.401487 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:34:13.401454 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:34:21.752278 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:34:21.752244 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:34:25.505456 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:34:25.505424 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:34:26.234361 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:34:26.234327 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:34:51.270568 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:34:51.270528 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6stkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:34:51.271228 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:34:51.271188 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" podUID="c6669c77-7d2c-48ba-829f-85ce6e6f9f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:34:55.704164 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:34:55.704131 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:34:55.900433 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:34:55.900345 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:34:56.234901 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:34:56.234863 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:34:59.654707 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:34:59.654670 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:35:06.274420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:35:06.273510 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6stkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:35:06.274420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:35:06.273597 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" podUID="c6669c77-7d2c-48ba-829f-85ce6e6f9f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:35:17.384362 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:35:17.384328 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:35:22.088138 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:35:22.088108 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:35:26.233655 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:35:26.233611 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:35:36.269611 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:35:36.269558 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-6stkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:35:36.270320 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:35:36.270267 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-6stkz" podUID="c6669c77-7d2c-48ba-829f-85ce6e6f9f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:35:56.235041 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:35:56.234953 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:36:03.424867 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:03.424795 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 18:36:03.427596 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:03.427574 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 18:36:05.686846 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.686813 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj"] Nov 03 18:36:05.687292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.687186 5805 cpu_manager.go:414] "containerMap: removing container" podUID="e8856532-061a-4f17-b49c-5e92228b7c02" containerName="discovery" Nov 03 18:36:05.687292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.687198 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8856532-061a-4f17-b49c-5e92228b7c02" containerName="discovery" Nov 03 18:36:05.687292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.687217 5805 cpu_manager.go:414] "containerMap: removing container" podUID="f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4" containerName="istio-proxy" Nov 03 18:36:05.687292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.687222 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4" containerName="istio-proxy" Nov 03 18:36:05.687292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.687230 5805 cpu_manager.go:414] "containerMap: removing container" podUID="e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" containerName="main" Nov 03 18:36:05.687292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.687235 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" containerName="main" Nov 03 18:36:05.687292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.687246 5805 cpu_manager.go:414] "containerMap: removing container" podUID="7238eabc-b932-49ce-aece-f9498937ecad" containerName="revoke-keys" Nov 03 18:36:05.687292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.687252 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7238eabc-b932-49ce-aece-f9498937ecad" containerName="revoke-keys" Nov 03 18:36:05.687292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.687268 5805 cpu_manager.go:414] "containerMap: removing container" podUID="60dc8570-85f6-4133-9567-5e99139b44e7" containerName="manager" Nov 03 18:36:05.687292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.687272 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60dc8570-85f6-4133-9567-5e99139b44e7" containerName="manager" Nov 03 18:36:05.687292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.687280 5805 cpu_manager.go:414] "containerMap: removing container" podUID="ec4a7156-46e6-4274-9794-e6d1f14aec3f" containerName="main" Nov 03 18:36:05.687292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.687284 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec4a7156-46e6-4274-9794-e6d1f14aec3f" containerName="main" Nov 03 18:36:05.687677 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.687359 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="7238eabc-b932-49ce-aece-f9498937ecad" containerName="revoke-keys" Nov 03 18:36:05.687677 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.687372 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="60dc8570-85f6-4133-9567-5e99139b44e7" containerName="manager" Nov 03 18:36:05.687677 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.687378 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec4a7156-46e6-4274-9794-e6d1f14aec3f" containerName="main" Nov 03 18:36:05.687677 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.687387 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8856532-061a-4f17-b49c-5e92228b7c02" containerName="discovery" Nov 03 18:36:05.687677 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.687405 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="f745c4ba-4f19-485f-9c87-3f5f2ab6ecb4" containerName="istio-proxy" Nov 03 18:36:05.687677 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.687410 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0aaaea9-8307-44e7-95d9-d83f0a2b6aa2" containerName="main" Nov 03 18:36:05.690555 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.690538 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj" Nov 03 18:36:05.746175 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.746141 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj"] Nov 03 18:36:05.796357 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.796324 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 18:36:05.796357 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.796324 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 18:36:05.796575 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.796365 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tr54c\"" Nov 03 18:36:05.817623 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.817592 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-maas-api-tls\") pod \"maas-api-7f85bcb458-9bkmj\" (UID: \"97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e\") " pod="odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj" Nov 03 18:36:05.817738 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.817644 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl4cq\" (UniqueName: \"kubernetes.io/projected/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-kube-api-access-bl4cq\") pod \"maas-api-7f85bcb458-9bkmj\" (UID: \"97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e\") " pod="odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj" Nov 03 18:36:05.817738 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.817702 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-metrics-certs\") pod \"maas-api-7f85bcb458-9bkmj\" (UID: \"97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e\") " pod="odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj" Nov 03 18:36:05.817813 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.817739 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-openshift-service-ca\") pod \"maas-api-7f85bcb458-9bkmj\" (UID: \"97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e\") " pod="odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj" Nov 03 18:36:05.918387 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.918358 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-openshift-service-ca\") pod \"maas-api-7f85bcb458-9bkmj\" (UID: \"97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e\") " pod="odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj" Nov 03 18:36:05.918545 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.918477 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-maas-api-tls\") pod \"maas-api-7f85bcb458-9bkmj\" (UID: \"97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e\") " pod="odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj" Nov 03 18:36:05.918545 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.918541 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl4cq\" (UniqueName: \"kubernetes.io/projected/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-kube-api-access-bl4cq\") pod \"maas-api-7f85bcb458-9bkmj\" (UID: \"97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e\") " pod="odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj" Nov 03 18:36:05.918634 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.918579 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-metrics-certs\") pod \"maas-api-7f85bcb458-9bkmj\" (UID: \"97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e\") " pod="odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj" Nov 03 18:36:05.919115 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.919093 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-openshift-service-ca\") pod \"maas-api-7f85bcb458-9bkmj\" (UID: \"97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e\") " pod="odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj" Nov 03 18:36:05.920894 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.920867 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-metrics-certs\") pod \"maas-api-7f85bcb458-9bkmj\" (UID: \"97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e\") " pod="odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj" Nov 03 18:36:05.921037 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.921018 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-maas-api-tls\") pod \"maas-api-7f85bcb458-9bkmj\" (UID: \"97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e\") " pod="odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj" Nov 03 18:36:05.980882 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:05.980820 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl4cq\" (UniqueName: \"kubernetes.io/projected/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-kube-api-access-bl4cq\") pod \"maas-api-7f85bcb458-9bkmj\" (UID: \"97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e\") " pod="odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj" Nov 03 18:36:06.003716 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:06.003688 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj" Nov 03 18:36:06.379747 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:06.379723 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj"] Nov 03 18:36:06.381311 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:36:06.381286 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a2e2d4_5fd7_4ef6_8b9e_07322c626e6e.slice/crio-2947ad0ebc1b3ea5c6c2cf0949dad98b652562ca9cedf72c59e75ff8a00d2f7b WatchSource:0}: Error finding container 2947ad0ebc1b3ea5c6c2cf0949dad98b652562ca9cedf72c59e75ff8a00d2f7b: Status 404 returned error can't find the container with id 2947ad0ebc1b3ea5c6c2cf0949dad98b652562ca9cedf72c59e75ff8a00d2f7b Nov 03 18:36:06.382535 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:06.382517 5805 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:36:07.336386 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:07.336358 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj" event={"ID":"97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e","Type":"ContainerStarted","Data":"2947ad0ebc1b3ea5c6c2cf0949dad98b652562ca9cedf72c59e75ff8a00d2f7b"} Nov 03 18:36:08.343470 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:08.343432 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj" event={"ID":"97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e","Type":"ContainerStarted","Data":"6af17346f86b69d05240441c654dac2f933f6ed54ba3ab819838073a985b5161"} Nov 03 18:36:08.343872 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:08.343523 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj" Nov 03 18:36:08.466196 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:08.466154 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj" podStartSLOduration=2.534268735 podStartE2EDuration="3.466141947s" totalImagesPullingTime="931.873212ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:36:08.464369313 +0000 UTC m=+3005.597998323" watchObservedRunningTime="2026-11-03 18:36:08.466141947 +0000 UTC m=+3005.599770957" Nov 03 18:36:12.419248 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:12.419212 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:36:13.933847 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:13.933814 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj"] Nov 03 18:36:13.934280 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:13.933999 5805 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj" podUID="97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e" containerName="maas-api" containerID="cri-o://6af17346f86b69d05240441c654dac2f933f6ed54ba3ab819838073a985b5161" gracePeriod=30 Nov 03 18:36:13.939543 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:13.939523 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj" Nov 03 18:36:13.985591 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:13.985564 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d477cc994-l78tq"] Nov 03 18:36:13.989068 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:13.989046 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d477cc994-l78tq" Nov 03 18:36:13.997859 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:13.997835 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a65876c-0ed2-40c3-821a-deaab738c514-metrics-certs\") pod \"maas-api-6d477cc994-l78tq\" (UID: \"5a65876c-0ed2-40c3-821a-deaab738c514\") " pod="odh-ai-gateway-infra/maas-api-6d477cc994-l78tq" Nov 03 18:36:13.997976 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:13.997864 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a65876c-0ed2-40c3-821a-deaab738c514-openshift-service-ca\") pod \"maas-api-6d477cc994-l78tq\" (UID: \"5a65876c-0ed2-40c3-821a-deaab738c514\") " pod="odh-ai-gateway-infra/maas-api-6d477cc994-l78tq" Nov 03 18:36:13.997976 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:13.997895 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5grgv\" (UniqueName: \"kubernetes.io/projected/5a65876c-0ed2-40c3-821a-deaab738c514-kube-api-access-5grgv\") pod \"maas-api-6d477cc994-l78tq\" (UID: \"5a65876c-0ed2-40c3-821a-deaab738c514\") " pod="odh-ai-gateway-infra/maas-api-6d477cc994-l78tq" Nov 03 18:36:13.998097 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:13.998070 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a65876c-0ed2-40c3-821a-deaab738c514-maas-api-tls\") pod \"maas-api-6d477cc994-l78tq\" (UID: \"5a65876c-0ed2-40c3-821a-deaab738c514\") " pod="odh-ai-gateway-infra/maas-api-6d477cc994-l78tq" Nov 03 18:36:14.060736 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.060708 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d477cc994-l78tq"] Nov 03 18:36:14.079428 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.079384 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:36:14.098762 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.098726 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a65876c-0ed2-40c3-821a-deaab738c514-metrics-certs\") pod \"maas-api-6d477cc994-l78tq\" (UID: \"5a65876c-0ed2-40c3-821a-deaab738c514\") " pod="odh-ai-gateway-infra/maas-api-6d477cc994-l78tq" Nov 03 18:36:14.098762 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.098764 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a65876c-0ed2-40c3-821a-deaab738c514-openshift-service-ca\") pod \"maas-api-6d477cc994-l78tq\" (UID: \"5a65876c-0ed2-40c3-821a-deaab738c514\") " pod="odh-ai-gateway-infra/maas-api-6d477cc994-l78tq" Nov 03 18:36:14.098987 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.098813 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5grgv\" (UniqueName: \"kubernetes.io/projected/5a65876c-0ed2-40c3-821a-deaab738c514-kube-api-access-5grgv\") pod \"maas-api-6d477cc994-l78tq\" (UID: \"5a65876c-0ed2-40c3-821a-deaab738c514\") " pod="odh-ai-gateway-infra/maas-api-6d477cc994-l78tq" Nov 03 18:36:14.098987 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.098887 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a65876c-0ed2-40c3-821a-deaab738c514-maas-api-tls\") pod \"maas-api-6d477cc994-l78tq\" (UID: \"5a65876c-0ed2-40c3-821a-deaab738c514\") " pod="odh-ai-gateway-infra/maas-api-6d477cc994-l78tq" Nov 03 18:36:14.099502 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.099476 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a65876c-0ed2-40c3-821a-deaab738c514-openshift-service-ca\") pod \"maas-api-6d477cc994-l78tq\" (UID: \"5a65876c-0ed2-40c3-821a-deaab738c514\") " pod="odh-ai-gateway-infra/maas-api-6d477cc994-l78tq" Nov 03 18:36:14.101201 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.101171 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a65876c-0ed2-40c3-821a-deaab738c514-maas-api-tls\") pod \"maas-api-6d477cc994-l78tq\" (UID: \"5a65876c-0ed2-40c3-821a-deaab738c514\") " pod="odh-ai-gateway-infra/maas-api-6d477cc994-l78tq" Nov 03 18:36:14.101312 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.101210 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a65876c-0ed2-40c3-821a-deaab738c514-metrics-certs\") pod \"maas-api-6d477cc994-l78tq\" (UID: \"5a65876c-0ed2-40c3-821a-deaab738c514\") " pod="odh-ai-gateway-infra/maas-api-6d477cc994-l78tq" Nov 03 18:36:14.161515 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.161486 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5grgv\" (UniqueName: \"kubernetes.io/projected/5a65876c-0ed2-40c3-821a-deaab738c514-kube-api-access-5grgv\") pod \"maas-api-6d477cc994-l78tq\" (UID: \"5a65876c-0ed2-40c3-821a-deaab738c514\") " pod="odh-ai-gateway-infra/maas-api-6d477cc994-l78tq" Nov 03 18:36:14.186710 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.186660 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj" Nov 03 18:36:14.199765 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.199745 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-kube-api-access-bl4cq\" (UniqueName: \"kubernetes.io/projected/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-kube-api-access-bl4cq\") pod \"97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e\" (UID: \"97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e\") " Nov 03 18:36:14.199851 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.199788 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-maas-api-tls\") pod \"97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e\" (UID: \"97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e\") " Nov 03 18:36:14.199851 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.199841 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-openshift-service-ca\") pod \"97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e\" (UID: \"97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e\") " Nov 03 18:36:14.199926 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.199903 5805 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-metrics-certs\") pod \"97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e\" (UID: \"97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e\") " Nov 03 18:36:14.200200 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.200177 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-openshift-service-ca" pod "97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e" (UID: "97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:36:14.201622 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.201597 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-kube-api-access-bl4cq" pod "97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e" (UID: "97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e"). InnerVolumeSpecName "kube-api-access-bl4cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:36:14.201781 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.201760 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-metrics-certs" pod "97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e" (UID: "97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:36:14.201852 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.201826 5805 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-maas-api-tls" pod "97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e" (UID: "97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:36:14.301033 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.301003 5805 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl4cq\" (UniqueName: \"kubernetes.io/projected/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-kube-api-access-bl4cq\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:36:14.301033 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.301027 5805 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-maas-api-tls\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:36:14.301033 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.301038 5805 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-openshift-service-ca\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:36:14.301236 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.301048 5805 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e-metrics-certs\") on node \"ip-10-0-137-164.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 18:36:14.341313 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.341288 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d477cc994-l78tq" Nov 03 18:36:14.372297 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.372273 5805 generic.go:356] "Generic (PLEG): container finished" podID="97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e" containerID="6af17346f86b69d05240441c654dac2f933f6ed54ba3ab819838073a985b5161" exitCode=0 Nov 03 18:36:14.372413 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.372358 5805 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj" Nov 03 18:36:14.372413 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.372359 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj" event={"ID":"97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e","Type":"ContainerDied","Data":"6af17346f86b69d05240441c654dac2f933f6ed54ba3ab819838073a985b5161"} Nov 03 18:36:14.372494 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.372451 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj" event={"ID":"97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e","Type":"ContainerDied","Data":"2947ad0ebc1b3ea5c6c2cf0949dad98b652562ca9cedf72c59e75ff8a00d2f7b"} Nov 03 18:36:14.372494 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.372475 5805 scope.go:122] "RemoveContainer" containerID="6af17346f86b69d05240441c654dac2f933f6ed54ba3ab819838073a985b5161" Nov 03 18:36:14.384572 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.384545 5805 scope.go:122] "RemoveContainer" containerID="6af17346f86b69d05240441c654dac2f933f6ed54ba3ab819838073a985b5161" Nov 03 18:36:14.384935 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:36:14.384917 5805 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6af17346f86b69d05240441c654dac2f933f6ed54ba3ab819838073a985b5161\": container with ID starting with 6af17346f86b69d05240441c654dac2f933f6ed54ba3ab819838073a985b5161 not found: ID does not exist" containerID="6af17346f86b69d05240441c654dac2f933f6ed54ba3ab819838073a985b5161" Nov 03 18:36:14.385005 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.384943 5805 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6af17346f86b69d05240441c654dac2f933f6ed54ba3ab819838073a985b5161"} err="failed to get container status \"6af17346f86b69d05240441c654dac2f933f6ed54ba3ab819838073a985b5161\": rpc error: code = NotFound desc = could not find container \"6af17346f86b69d05240441c654dac2f933f6ed54ba3ab819838073a985b5161\": container with ID starting with 6af17346f86b69d05240441c654dac2f933f6ed54ba3ab819838073a985b5161 not found: ID does not exist" Nov 03 18:36:14.534167 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.534094 5805 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj"] Nov 03 18:36:14.538200 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.538173 5805 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f85bcb458-9bkmj"] Nov 03 18:36:14.724292 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.724269 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d477cc994-l78tq"] Nov 03 18:36:14.726041 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:36:14.726015 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a65876c_0ed2_40c3_821a_deaab738c514.slice/crio-029e758293d0e222b638897e98114bcd7001477a254bf28414d875917aa0f736 WatchSource:0}: Error finding container 029e758293d0e222b638897e98114bcd7001477a254bf28414d875917aa0f736: Status 404 returned error can't find the container with id 029e758293d0e222b638897e98114bcd7001477a254bf28414d875917aa0f736 Nov 03 18:36:14.975113 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:14.975085 5805 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e" path="/var/lib/kubelet/pods/97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e/volumes" Nov 03 18:36:15.378915 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:15.378880 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d477cc994-l78tq" event={"ID":"5a65876c-0ed2-40c3-821a-deaab738c514","Type":"ContainerStarted","Data":"029e758293d0e222b638897e98114bcd7001477a254bf28414d875917aa0f736"} Nov 03 18:36:16.390709 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:16.390669 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d477cc994-l78tq" event={"ID":"5a65876c-0ed2-40c3-821a-deaab738c514","Type":"ContainerStarted","Data":"c786bc1d8c89637363d23cb4ec2a0ec45da5232dd66ec28ead7ba800b08ddccf"} Nov 03 18:36:16.391126 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:16.390840 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d477cc994-l78tq" Nov 03 18:36:16.513549 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:16.513492 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d477cc994-l78tq" podStartSLOduration=2.563284365 podStartE2EDuration="3.513472643s" totalImagesPullingTime="950.188278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:36:16.511980733 +0000 UTC m=+3013.645609763" watchObservedRunningTime="2026-11-03 18:36:16.513472643 +0000 UTC m=+3013.647101654" Nov 03 18:36:17.295362 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:17.295327 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:36:17.503194 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:17.503161 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:36:20.534828 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:20.534794 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:36:20.799132 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:20.799049 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:36:21.004456 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:21.004389 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:36:21.206233 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:21.206198 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:36:22.400688 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:22.400660 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d477cc994-l78tq" Nov 03 18:36:22.449341 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:22.449313 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:36:26.234333 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:36:26.234301 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:36:26.295128 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:26.295091 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:36:31.204096 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:31.204055 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:36:36.011138 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:36.011109 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:36:41.209663 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:41.209630 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:36:46.005422 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:46.005369 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:36:50.907192 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:50.907160 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:36:55.708199 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:55.708167 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:36:56.234702 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:36:56.234669 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:36:59.962716 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:36:59.962680 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:00.459031 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:37:00.458994 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:00.657817 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:37:00.657786 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:04.462002 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:37:04.461969 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:04.746177 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:37:04.746097 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:09.864525 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:37:09.864487 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:11.214074 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:37:11.214044 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:11.857871 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:37:11.857829 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:12.298526 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:37:12.298412 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:16.236988 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:37:16.236953 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:16.437667 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:37:16.437631 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:16.646077 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:37:16.646045 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:19.463681 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:37:19.463644 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:19.670464 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:37:19.670431 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:22.553661 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:37:22.553620 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:22.745023 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:37:22.744990 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:22.946341 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:37:22.946311 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:26.234090 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:37:26.234050 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:37:27.529015 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:37:27.528979 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:32.430445 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:37:32.430411 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:37.353021 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:37:37.352983 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:41.182625 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:37:41.182591 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:42.560993 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:37:42.560960 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:47.434070 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:37:47.434026 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:52.351447 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:37:52.351378 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:37:56.235491 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:37:56.235453 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:37:57.111432 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:37:57.111389 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:02.015303 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:38:02.015260 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:06.805115 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:38:06.805078 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:11.716774 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:38:11.716717 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:16.509824 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:38:16.509789 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:21.415758 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:38:21.415726 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:23.139859 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:38:23.139820 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:26.211634 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:38:26.211595 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:26.234346 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:38:26.234312 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:38:32.178160 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:38:32.178119 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:35.880823 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:38:35.880778 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:36.076928 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:38:36.076893 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:40.807227 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:38:40.807194 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:45.621935 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:38:45.621721 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:50.502823 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:38:50.502749 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:56.234214 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:38:56.234165 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:38:56.350560 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:38:56.350529 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:38:56.773306 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:38:56.773268 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:00.203194 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:39:00.203160 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:05.352441 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:39:05.352387 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:10.404182 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:39:10.404146 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:13.839242 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:39:13.839206 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:14.042386 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:39:14.042349 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:15.212868 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:39:15.212835 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:25.260753 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:39:25.260711 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:25.461461 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:39:25.461426 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:26.018066 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:39:26.018031 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:26.234930 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:39:26.234889 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:39:26.308628 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:39:26.308601 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:27.102615 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:39:27.102580 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:33.111558 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:39:33.111521 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:33.319442 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:39:33.319411 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:33.518555 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:39:33.518524 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:39:52.663421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:39:52.662526 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-jmn7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:39:52.663421 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:39:52.662619 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-jmn7m" podUID="3369b197-476f-4861-b6c3-ac9bfa5464d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:39:56.234621 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:39:56.234584 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:40:23.846477 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:40:23.846440 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:40:26.234725 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:40:26.234689 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:40:37.742973 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:40:37.742930 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:40:37.944840 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:40:37.944807 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:40:45.540419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:40:45.538510 5805 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vgzph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:40:45.540419 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:40:45.538596 5805 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vgzph" podUID="d685b092-b52b-4ca9-a114-dd71a3be6f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:40:54.816661 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:40:54.816628 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-sbsh4_654e1963-654e-44e1-b284-1cf0bb09216d/manager/0.log" Nov 03 18:40:55.132452 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:40:55.132362 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-zwb45_7c3ba816-8be5-4c56-8be8-6f7dae15d058/manager/0.log" Nov 03 18:40:56.099749 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:40:56.099715 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-q7657_707bc0f1-d6de-45c3-9783-dd6772ca19eb/server/0.log" Nov 03 18:40:56.234609 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:40:56.234573 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:40:56.578916 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:40:56.578883 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-b9mh2_f544ec2f-3048-4b83-a594-0c7a412c3076/proxy/0.log" Nov 03 18:40:57.665475 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:40:57.665449 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6d477cc994-l78tq_5a65876c-0ed2-40c3-821a-deaab738c514/maas-api/0.log" Nov 03 18:40:58.650563 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:40:58.650523 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-ff8wf_e1e3a570-a9f9-41c6-b5ab-51898a697a90/postgres/0.log" Nov 03 18:41:01.174725 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:41:01.174696 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-4k7pg_0c040730-05c1-4271-8c50-7ba7424d3205/kuadrant-console-plugin/0.log" Nov 03 18:41:03.461545 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:41:03.461516 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54df9ccd9f-v8bvm_a5aa44e3-5d93-4f17-869f-aa09922776c0/kube-auth-proxy/0.log" Nov 03 18:41:03.467488 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:41:03.467465 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 18:41:03.471469 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:41:03.471452 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jjq2h_6a74bba5-bc9c-4ae1-9754-8887c5bf23cc/ovn-acl-logging/0.log" Nov 03 18:41:04.130629 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:41:04.130598 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-6stkz_c6669c77-7d2c-48ba-829f-85ce6e6f9f0c/istio-proxy/0.log" Nov 03 18:41:05.440562 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:41:05.440528 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-jmn7m_3369b197-476f-4861-b6c3-ac9bfa5464d1/istio-proxy/0.log" Nov 03 18:41:06.194330 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:41:06.194303 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-vgzph_d685b092-b52b-4ca9-a114-dd71a3be6f40/istio-proxy/0.log" Nov 03 18:41:06.429181 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:41:06.429150 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d_bdccc585-8c1c-451d-bacd-3e5933cc4b06/istio-proxy/0.log" Nov 03 18:41:20.920901 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:41:20.920872 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-sbsh4_654e1963-654e-44e1-b284-1cf0bb09216d/manager/0.log" Nov 03 18:41:21.208373 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:41:21.208286 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-zwb45_7c3ba816-8be5-4c56-8be8-6f7dae15d058/manager/0.log" Nov 03 18:41:21.925288 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:41:21.925257 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-q7657_707bc0f1-d6de-45c3-9783-dd6772ca19eb/server/0.log" Nov 03 18:41:22.448796 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:41:22.448764 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-b9mh2_f544ec2f-3048-4b83-a594-0c7a412c3076/proxy/0.log" Nov 03 18:41:23.357600 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:41:23.357568 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6d477cc994-l78tq_5a65876c-0ed2-40c3-821a-deaab738c514/maas-api/0.log" Nov 03 18:41:24.198474 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:41:24.198435 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:41:24.327323 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:41:24.327270 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-ff8wf_e1e3a570-a9f9-41c6-b5ab-51898a697a90/postgres/0.log" Nov 03 18:41:26.234016 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:41:26.233978 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:41:26.438189 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:41:26.438152 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-4k7pg_0c040730-05c1-4271-8c50-7ba7424d3205/kuadrant-console-plugin/0.log" Nov 03 18:41:27.037908 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:41:27.037875 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:41:28.442139 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:41:28.442111 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54df9ccd9f-v8bvm_a5aa44e3-5d93-4f17-869f-aa09922776c0/kube-auth-proxy/0.log" Nov 03 18:41:29.124580 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:41:29.124548 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-6stkz_c6669c77-7d2c-48ba-829f-85ce6e6f9f0c/istio-proxy/0.log" Nov 03 18:41:30.413597 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:41:30.413571 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-jmn7m_3369b197-476f-4861-b6c3-ac9bfa5464d1/istio-proxy/0.log" Nov 03 18:41:31.149284 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:41:31.149253 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-vgzph_d685b092-b52b-4ca9-a114-dd71a3be6f40/istio-proxy/0.log" Nov 03 18:41:31.365372 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:41:31.365340 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-58fd47455f-4zh8d_bdccc585-8c1c-451d-bacd-3e5933cc4b06/istio-proxy/0.log" Nov 03 18:41:56.233991 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:41:56.233910 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:42:02.780318 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:42:02.780250 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:42:02.980166 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:42:02.980115 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:42:20.413572 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:42:20.413502 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:42:20.611693 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:42:20.611657 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:42:24.545612 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:42:24.545578 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:42:26.237023 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:42:26.236983 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:42:46.738283 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:42:46.738242 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:42:46.933512 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:42:46.933477 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:42:56.234164 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:42:56.234122 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor" Nov 03 18:42:56.340308 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:42:56.340276 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8zpkh_c527a13f-da23-4228-a048-4af962c36340/global-pull-secret-syncer/0.log" Nov 03 18:42:58.979840 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:42:58.979811 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" Nov 03 18:43:00.780538 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:00.780507 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-164.us-west-2.compute.internal_5df37dbe21c11f3ff8879a1693476b0b/haproxy/0.log" Nov 03 18:43:04.396057 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:04.396024 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:43:05.582482 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.582448 5805 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q"] Nov 03 18:43:05.582884 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.582867 5805 cpu_manager.go:414] "containerMap: removing container" podUID="97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e" containerName="maas-api" Nov 03 18:43:05.582884 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.582877 5805 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e" containerName="maas-api" Nov 03 18:43:05.582975 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.582950 5805 memory_manager.go:365] "RemoveStaleState removing state" podUID="97a2e2d4-5fd7-4ef6-8b9e-07322c626e6e" containerName="maas-api" Nov 03 18:43:05.586064 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.586039 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q" Nov 03 18:43:05.633142 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.633117 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:43:05.634219 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.634200 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k8k5h\"/\"kube-root-ca.crt\"" Nov 03 18:43:05.679752 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.679723 5805 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k8k5h\"/\"openshift-service-ca.crt\"" Nov 03 18:43:05.679929 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.679873 5805 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k8k5h\"/\"default-dockercfg-dhhjm\"" Nov 03 18:43:05.681845 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.681819 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4kzf\" (UniqueName: \"kubernetes.io/projected/88f00a19-c72d-4267-a359-8384d1b37ef4-kube-api-access-x4kzf\") pod \"perf-node-gather-daemonset-bc58q\" (UID: \"88f00a19-c72d-4267-a359-8384d1b37ef4\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q" Nov 03 18:43:05.681953 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.681861 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/88f00a19-c72d-4267-a359-8384d1b37ef4-proc\") pod \"perf-node-gather-daemonset-bc58q\" (UID: \"88f00a19-c72d-4267-a359-8384d1b37ef4\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q" Nov 03 18:43:05.682020 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.681949 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88f00a19-c72d-4267-a359-8384d1b37ef4-sys\") pod \"perf-node-gather-daemonset-bc58q\" (UID: \"88f00a19-c72d-4267-a359-8384d1b37ef4\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q" Nov 03 18:43:05.682020 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.682010 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/88f00a19-c72d-4267-a359-8384d1b37ef4-podres\") pod \"perf-node-gather-daemonset-bc58q\" (UID: \"88f00a19-c72d-4267-a359-8384d1b37ef4\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q" Nov 03 18:43:05.682139 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.682089 5805 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/88f00a19-c72d-4267-a359-8384d1b37ef4-lib-modules\") pod \"perf-node-gather-daemonset-bc58q\" (UID: \"88f00a19-c72d-4267-a359-8384d1b37ef4\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q" Nov 03 18:43:05.682289 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.682271 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q"] Nov 03 18:43:05.783564 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.783538 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/88f00a19-c72d-4267-a359-8384d1b37ef4-lib-modules\") pod \"perf-node-gather-daemonset-bc58q\" (UID: \"88f00a19-c72d-4267-a359-8384d1b37ef4\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q" Nov 03 18:43:05.783698 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.783648 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/88f00a19-c72d-4267-a359-8384d1b37ef4-lib-modules\") pod \"perf-node-gather-daemonset-bc58q\" (UID: \"88f00a19-c72d-4267-a359-8384d1b37ef4\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q" Nov 03 18:43:05.783698 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.783676 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4kzf\" (UniqueName: \"kubernetes.io/projected/88f00a19-c72d-4267-a359-8384d1b37ef4-kube-api-access-x4kzf\") pod \"perf-node-gather-daemonset-bc58q\" (UID: \"88f00a19-c72d-4267-a359-8384d1b37ef4\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q" Nov 03 18:43:05.783787 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.783706 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/88f00a19-c72d-4267-a359-8384d1b37ef4-proc\") pod \"perf-node-gather-daemonset-bc58q\" (UID: \"88f00a19-c72d-4267-a359-8384d1b37ef4\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q" Nov 03 18:43:05.783787 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.783727 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88f00a19-c72d-4267-a359-8384d1b37ef4-sys\") pod \"perf-node-gather-daemonset-bc58q\" (UID: \"88f00a19-c72d-4267-a359-8384d1b37ef4\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q" Nov 03 18:43:05.783787 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.783775 5805 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/88f00a19-c72d-4267-a359-8384d1b37ef4-podres\") pod \"perf-node-gather-daemonset-bc58q\" (UID: \"88f00a19-c72d-4267-a359-8384d1b37ef4\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q" Nov 03 18:43:05.783901 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.783790 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/88f00a19-c72d-4267-a359-8384d1b37ef4-proc\") pod \"perf-node-gather-daemonset-bc58q\" (UID: \"88f00a19-c72d-4267-a359-8384d1b37ef4\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q" Nov 03 18:43:05.783901 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.783860 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88f00a19-c72d-4267-a359-8384d1b37ef4-sys\") pod \"perf-node-gather-daemonset-bc58q\" (UID: \"88f00a19-c72d-4267-a359-8384d1b37ef4\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q" Nov 03 18:43:05.784018 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.783904 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/88f00a19-c72d-4267-a359-8384d1b37ef4-podres\") pod \"perf-node-gather-daemonset-bc58q\" (UID: \"88f00a19-c72d-4267-a359-8384d1b37ef4\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q" Nov 03 18:43:05.833737 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.833673 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:43:05.840086 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.840061 5805 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4kzf\" (UniqueName: \"kubernetes.io/projected/88f00a19-c72d-4267-a359-8384d1b37ef4-kube-api-access-x4kzf\") pod \"perf-node-gather-daemonset-bc58q\" (UID: \"88f00a19-c72d-4267-a359-8384d1b37ef4\") " pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q" Nov 03 18:43:05.892049 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.892004 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-4k7pg_0c040730-05c1-4271-8c50-7ba7424d3205/kuadrant-console-plugin/0.log" Nov 03 18:43:05.898171 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:05.898142 5805 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q" Nov 03 18:43:06.270133 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:06.270054 5805 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q"] Nov 03 18:43:06.277430 ip-10-0-137-164 kubenswrapper[5805]: W1103 18:43:06.277386 5805 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod88f00a19_c72d_4267_a359_8384d1b37ef4.slice/crio-ca26f524abf8e33f8027b639b3b884507f84eae33b8e152aec650516c375a476 WatchSource:0}: Error finding container ca26f524abf8e33f8027b639b3b884507f84eae33b8e152aec650516c375a476: Status 404 returned error can't find the container with id ca26f524abf8e33f8027b639b3b884507f84eae33b8e152aec650516c375a476 Nov 03 18:43:06.458364 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:06.458325 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q" event={"ID":"88f00a19-c72d-4267-a359-8384d1b37ef4","Type":"ContainerStarted","Data":"67e20c09d426a72c3cf68e1f8e6fbb593ed263697f21f92863dcfed151aba319"} Nov 03 18:43:06.458552 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:06.458428 5805 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q" event={"ID":"88f00a19-c72d-4267-a359-8384d1b37ef4","Type":"ContainerStarted","Data":"ca26f524abf8e33f8027b639b3b884507f84eae33b8e152aec650516c375a476"} Nov 03 18:43:06.458552 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:06.458506 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q" Nov 03 18:43:06.568189 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:06.568136 5805 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q" podStartSLOduration=1.5681229700000001 podStartE2EDuration="1.56812297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:43:06.566663385 +0000 UTC m=+3423.700292395" watchObservedRunningTime="2026-11-03 18:43:06.56812297 +0000 UTC m=+3423.701751979" Nov 03 18:43:09.572515 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:09.572491 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c186bca9-0d99-4c9f-a1d9-c8316bd4155f/alertmanager/0.log" Nov 03 18:43:09.964074 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:09.964003 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c186bca9-0d99-4c9f-a1d9-c8316bd4155f/config-reloader/0.log" Nov 03 18:43:10.357341 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:10.357316 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c186bca9-0d99-4c9f-a1d9-c8316bd4155f/kube-rbac-proxy-web/0.log" Nov 03 18:43:10.750420 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:10.750323 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c186bca9-0d99-4c9f-a1d9-c8316bd4155f/kube-rbac-proxy/0.log" Nov 03 18:43:11.143720 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:11.143688 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c186bca9-0d99-4c9f-a1d9-c8316bd4155f/kube-rbac-proxy-metric/0.log" Nov 03 18:43:11.536512 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:11.536441 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c186bca9-0d99-4c9f-a1d9-c8316bd4155f/prom-label-proxy/0.log" Nov 03 18:43:11.929893 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:11.929848 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c186bca9-0d99-4c9f-a1d9-c8316bd4155f/init-config-reloader/0.log" Nov 03 18:43:12.473696 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:12.473670 5805 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-k8k5h/perf-node-gather-daemonset-bc58q" Nov 03 18:43:12.751134 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:12.751052 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2z6hc_83572d76-31ae-42e0-b7d4-acc08df84a9c/kube-state-metrics/0.log" Nov 03 18:43:13.139944 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:13.139917 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2z6hc_83572d76-31ae-42e0-b7d4-acc08df84a9c/kube-rbac-proxy-main/0.log" Nov 03 18:43:13.532247 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:13.532174 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2z6hc_83572d76-31ae-42e0-b7d4-acc08df84a9c/kube-rbac-proxy-self/0.log" Nov 03 18:43:13.927354 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:13.927316 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-687b6669ff-vc8sm_d82f08c7-9da6-4969-858f-c8e5b140d27c/metrics-server/0.log" Nov 03 18:43:17.136663 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:17.136634 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sf77n_a8abced1-c904-4e84-b62e-96290db6fa8e/node-exporter/0.log" Nov 03 18:43:17.537956 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:17.537868 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sf77n_a8abced1-c904-4e84-b62e-96290db6fa8e/kube-rbac-proxy/0.log" Nov 03 18:43:17.932049 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:17.932020 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sf77n_a8abced1-c904-4e84-b62e-96290db6fa8e/init-textfile/0.log" Nov 03 18:43:18.326787 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:18.326757 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-c5q4r_78a8038b-489c-44f5-827f-d4aaebca42f8/kube-rbac-proxy-main/0.log" Nov 03 18:43:18.718556 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:18.718469 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-c5q4r_78a8038b-489c-44f5-827f-d4aaebca42f8/kube-rbac-proxy-self/0.log" Nov 03 18:43:19.111619 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:19.111558 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-c5q4r_78a8038b-489c-44f5-827f-d4aaebca42f8/openshift-state-metrics/0.log" Nov 03 18:43:19.514012 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:19.513932 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5440d502-9859-436b-bba4-acb92ec3282e/prometheus/0.log" Nov 03 18:43:19.905727 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:19.905699 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5440d502-9859-436b-bba4-acb92ec3282e/config-reloader/0.log" Nov 03 18:43:20.302204 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:20.302120 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5440d502-9859-436b-bba4-acb92ec3282e/thanos-sidecar/0.log" Nov 03 18:43:20.698298 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:20.698248 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5440d502-9859-436b-bba4-acb92ec3282e/kube-rbac-proxy-web/0.log" Nov 03 18:43:21.108123 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:21.108091 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5440d502-9859-436b-bba4-acb92ec3282e/kube-rbac-proxy/0.log" Nov 03 18:43:21.501101 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:21.501012 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5440d502-9859-436b-bba4-acb92ec3282e/kube-rbac-proxy-thanos/0.log" Nov 03 18:43:21.893357 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:21.893332 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5440d502-9859-436b-bba4-acb92ec3282e/init-config-reloader/0.log" Nov 03 18:43:23.509449 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:23.509421 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67d8bb587d-lcgff_c5890bb8-3e6a-4250-8ccb-50771cd112c0/telemeter-client/0.log" Nov 03 18:43:23.900505 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:23.900478 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67d8bb587d-lcgff_c5890bb8-3e6a-4250-8ccb-50771cd112c0/reload/0.log" Nov 03 18:43:24.291633 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:24.291564 5805 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-67d8bb587d-lcgff_c5890bb8-3e6a-4250-8ccb-50771cd112c0/kube-rbac-proxy/0.log" Nov 03 18:43:24.885314 ip-10-0-137-164 kubenswrapper[5805]: I1103 18:43:24.885280 5805 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8b7gg_fa56a595-a9ef-431d-b176-41cb1cf382d7/konnectivity-agent/0.log" limit=1024 Nov 03 18:43:26.235247 ip-10-0-137-164 kubenswrapper[5805]: E1103 18:43:26.235204 5805 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rbt65" containerName="extractor"