Aug 04 17:52:24.192953 ip-10-0-137-173 systemd[1]: Starting Kubernetes Kubelet... Aug 04 17:52:24.599237 ip-10-0-137-173 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 17:52:24.599237 ip-10-0-137-173 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 04 17:52:24.599237 ip-10-0-137-173 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 17:52:24.599237 ip-10-0-137-173 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 04 17:52:24.599237 ip-10-0-137-173 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 17:52:24.599947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.599258 2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 04 17:52:24.606072 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606044 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 17:52:24.606072 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606062 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 17:52:24.606072 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606066 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 17:52:24.606072 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606070 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 17:52:24.606072 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606073 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 17:52:24.606072 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606076 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 17:52:24.606072 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606079 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 17:52:24.606072 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606082 2572 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 17:52:24.606343 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606085 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 17:52:24.606343 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606088 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 17:52:24.606343 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606093 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 17:52:24.606343 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606098 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 17:52:24.606343 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606101 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 17:52:24.606343 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606104 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 17:52:24.606343 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606106 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 17:52:24.606343 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606109 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 17:52:24.606343 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606112 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 17:52:24.606343 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606114 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 17:52:24.606343 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606117 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 17:52:24.606343 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606119 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 17:52:24.606343 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606122 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 17:52:24.606343 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606125 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 17:52:24.606343 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606127 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 17:52:24.606343 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606129 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 17:52:24.606343 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606132 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 17:52:24.606343 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606134 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 17:52:24.606343 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606138 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 17:52:24.606343 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606140 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 17:52:24.606842 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606143 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 17:52:24.606842 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606145 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 17:52:24.606842 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606148 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 17:52:24.606842 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606150 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 17:52:24.606842 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606153 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 17:52:24.606842 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606155 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 17:52:24.606842 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606158 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 17:52:24.606842 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606160 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 17:52:24.606842 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606163 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 17:52:24.606842 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606165 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 17:52:24.606842 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606168 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 17:52:24.606842 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606170 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 17:52:24.606842 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606173 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 17:52:24.606842 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606176 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 17:52:24.606842 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606178 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 17:52:24.606842 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606181 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 17:52:24.606842 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606183 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 17:52:24.606842 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606186 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 17:52:24.606842 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606188 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 17:52:24.606842 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606191 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 17:52:24.607324 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606193 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 17:52:24.607324 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606197 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 17:52:24.607324 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606201 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 17:52:24.607324 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606203 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 17:52:24.607324 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606206 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 17:52:24.607324 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606208 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 17:52:24.607324 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606210 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 17:52:24.607324 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606213 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 17:52:24.607324 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606217 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 17:52:24.607324 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606220 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 17:52:24.607324 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606223 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 17:52:24.607324 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606226 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 17:52:24.607324 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606229 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 17:52:24.607324 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606232 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 17:52:24.607324 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606234 2572 feature_gate.go:328] unrecognized feature gate: Example Aug 04 17:52:24.607324 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606237 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 17:52:24.607324 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606240 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 17:52:24.607324 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606243 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 17:52:24.607324 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606245 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 17:52:24.607324 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606248 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 17:52:24.607828 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606251 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 17:52:24.607828 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606253 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 17:52:24.607828 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606256 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 17:52:24.607828 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606259 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 17:52:24.607828 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606261 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 17:52:24.607828 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606264 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 17:52:24.607828 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606267 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 17:52:24.607828 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606269 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 17:52:24.607828 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606272 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 17:52:24.607828 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606275 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 17:52:24.607828 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606277 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 17:52:24.607828 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606282 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 17:52:24.607828 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606286 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 17:52:24.607828 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606289 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 17:52:24.607828 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606291 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 17:52:24.607828 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606295 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 17:52:24.607828 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606297 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 17:52:24.607828 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606300 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 17:52:24.607828 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606302 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 17:52:24.608280 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606305 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 17:52:24.608280 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606307 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 17:52:24.608280 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606310 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 17:52:24.608280 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606312 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 17:52:24.608280 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606315 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 17:52:24.608280 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606317 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 17:52:24.608280 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606320 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 17:52:24.608280 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606323 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 17:52:24.608280 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606325 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 17:52:24.608280 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606328 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 17:52:24.608280 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606331 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 17:52:24.608280 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.606334 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 17:52:24.608280 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607074 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 17:52:24.608280 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607081 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 17:52:24.608280 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607084 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 17:52:24.608280 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607087 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 17:52:24.608280 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607089 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 17:52:24.608280 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607093 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 17:52:24.608280 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607096 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 17:52:24.608280 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607099 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 17:52:24.608769 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607101 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 17:52:24.608769 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607104 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 17:52:24.608769 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607107 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 17:52:24.608769 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607109 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 17:52:24.608769 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607111 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 17:52:24.608769 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607114 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 17:52:24.608769 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607116 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 17:52:24.608769 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607119 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 17:52:24.608769 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607121 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 17:52:24.608769 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607125 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 17:52:24.608769 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607127 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 17:52:24.608769 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607130 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 17:52:24.608769 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607132 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 17:52:24.608769 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607134 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 17:52:24.608769 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607137 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 17:52:24.608769 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607139 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 17:52:24.608769 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607141 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 17:52:24.608769 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607144 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 17:52:24.608769 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607146 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 17:52:24.608769 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607149 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 17:52:24.609281 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607151 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 17:52:24.609281 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607154 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 17:52:24.609281 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607156 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 17:52:24.609281 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607159 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 17:52:24.609281 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607161 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 17:52:24.609281 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607164 2572 feature_gate.go:328] unrecognized feature gate: Example Aug 04 17:52:24.609281 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607166 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 17:52:24.609281 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607169 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 17:52:24.609281 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607172 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 17:52:24.609281 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607174 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 17:52:24.609281 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607177 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 17:52:24.609281 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607180 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 17:52:24.609281 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607182 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 17:52:24.609281 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607185 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 17:52:24.609281 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607187 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 17:52:24.609281 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607190 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 17:52:24.609281 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607192 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 17:52:24.609281 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607195 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 17:52:24.609281 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607197 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 17:52:24.609281 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607200 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 17:52:24.609792 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607203 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 17:52:24.609792 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607205 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 17:52:24.609792 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607208 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 17:52:24.609792 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607210 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 17:52:24.609792 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607214 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 17:52:24.609792 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607217 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 17:52:24.609792 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607222 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 17:52:24.609792 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607225 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 17:52:24.609792 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607228 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 17:52:24.609792 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607231 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 17:52:24.609792 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607235 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 17:52:24.609792 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607238 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 17:52:24.609792 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607241 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 17:52:24.609792 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607244 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 17:52:24.609792 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607247 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 17:52:24.609792 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607249 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 17:52:24.609792 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607252 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 17:52:24.609792 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607254 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 17:52:24.610266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607257 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 17:52:24.610266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607259 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 17:52:24.610266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607262 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 17:52:24.610266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607264 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 17:52:24.610266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607266 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 17:52:24.610266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607269 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 17:52:24.610266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607271 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 17:52:24.610266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607274 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 17:52:24.610266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607276 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 17:52:24.610266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607279 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 17:52:24.610266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607281 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 17:52:24.610266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607285 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 17:52:24.610266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607287 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 17:52:24.610266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607289 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 17:52:24.610266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607292 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 17:52:24.610266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607294 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 17:52:24.610266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607297 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 17:52:24.610266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607299 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 17:52:24.610266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607302 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 17:52:24.610266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607304 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 17:52:24.610778 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607307 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 17:52:24.610778 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607310 2572 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 17:52:24.610778 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607312 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 17:52:24.610778 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607314 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 17:52:24.610778 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607317 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 17:52:24.610778 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607319 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 17:52:24.610778 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607321 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 17:52:24.610778 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607325 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 17:52:24.610778 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607328 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 17:52:24.610778 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607330 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 17:52:24.610778 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607333 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 17:52:24.610778 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607336 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 17:52:24.610778 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607338 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 17:52:24.610778 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607716 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 17:52:24.610778 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607721 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 17:52:24.610778 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607734 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 17:52:24.610778 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607737 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 17:52:24.610778 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607739 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 17:52:24.610778 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607743 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 17:52:24.610778 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607745 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 17:52:24.611266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607748 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 17:52:24.611266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607751 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 17:52:24.611266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607753 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 17:52:24.611266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607756 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 17:52:24.611266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607758 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 17:52:24.611266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607761 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 17:52:24.611266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607763 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 17:52:24.611266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607766 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 17:52:24.611266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607768 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 17:52:24.611266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607772 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 17:52:24.611266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607774 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 17:52:24.611266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607777 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 17:52:24.611266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607779 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 17:52:24.611266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607782 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 17:52:24.611266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607784 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 17:52:24.611266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607787 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 17:52:24.611266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607789 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 17:52:24.611266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607791 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 17:52:24.611266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607794 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 17:52:24.611266 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607796 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 17:52:24.611765 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607800 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 17:52:24.611765 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607802 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 17:52:24.611765 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607805 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 17:52:24.611765 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607807 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 17:52:24.611765 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607810 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 17:52:24.611765 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607813 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 17:52:24.611765 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607815 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 17:52:24.611765 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607818 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 17:52:24.611765 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607820 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 17:52:24.611765 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607822 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 17:52:24.611765 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607825 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 17:52:24.611765 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607827 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 17:52:24.611765 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607830 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 17:52:24.611765 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607833 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 17:52:24.611765 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607835 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 17:52:24.611765 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607838 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 17:52:24.611765 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607841 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 17:52:24.611765 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607843 2572 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 17:52:24.611765 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607846 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 17:52:24.611765 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607848 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 17:52:24.612237 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607850 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 17:52:24.612237 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607853 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 17:52:24.612237 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607858 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 17:52:24.612237 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607861 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 17:52:24.612237 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607863 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 17:52:24.612237 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607866 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 17:52:24.612237 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607868 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 17:52:24.612237 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607871 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 17:52:24.612237 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607873 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 17:52:24.612237 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607875 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 17:52:24.612237 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607878 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 17:52:24.612237 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607881 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 17:52:24.612237 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607886 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 17:52:24.612237 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607889 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 17:52:24.612237 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607891 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 17:52:24.612237 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607895 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 17:52:24.612237 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607899 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 17:52:24.612237 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607902 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 17:52:24.612237 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607906 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 17:52:24.612711 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607909 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 17:52:24.612711 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607913 2572 feature_gate.go:328] unrecognized feature gate: Example Aug 04 17:52:24.612711 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607915 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 17:52:24.612711 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607918 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 17:52:24.612711 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607921 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 17:52:24.612711 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607925 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 17:52:24.612711 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607928 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 17:52:24.612711 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607931 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 17:52:24.612711 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607934 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 17:52:24.612711 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607937 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 17:52:24.612711 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607939 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 17:52:24.612711 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607942 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 17:52:24.612711 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607945 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 17:52:24.612711 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607948 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 17:52:24.612711 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607950 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 17:52:24.612711 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607952 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 17:52:24.612711 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607957 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 17:52:24.612711 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607960 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 17:52:24.612711 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607963 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 17:52:24.613167 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607965 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 17:52:24.613167 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607967 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 17:52:24.613167 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607970 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 17:52:24.613167 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607972 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 17:52:24.613167 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607975 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 17:52:24.613167 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607977 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 17:52:24.613167 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607981 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 17:52:24.613167 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607984 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 17:52:24.613167 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607987 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 17:52:24.613167 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607989 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 17:52:24.613167 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607992 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 17:52:24.613167 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607994 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 17:52:24.613167 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607997 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 17:52:24.613167 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.607999 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 17:52:24.613167 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608085 2572 flags.go:64] FLAG: --address="0.0.0.0" Aug 04 17:52:24.613167 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608091 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 04 17:52:24.613167 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608098 2572 flags.go:64] FLAG: --anonymous-auth="true" Aug 04 17:52:24.613167 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608102 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 04 17:52:24.613167 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608107 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 04 17:52:24.613167 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608110 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 04 17:52:24.613167 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608114 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 04 17:52:24.613713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608118 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 04 17:52:24.613713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608121 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 04 17:52:24.613713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608124 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 04 17:52:24.613713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608127 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 04 17:52:24.613713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608131 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 04 17:52:24.613713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608134 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 04 17:52:24.613713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608136 2572 flags.go:64] FLAG: --cgroup-root="" Aug 04 17:52:24.613713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608139 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 04 17:52:24.613713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608142 2572 flags.go:64] FLAG: --client-ca-file="" Aug 04 17:52:24.613713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608150 2572 flags.go:64] FLAG: --cloud-provider="external" Aug 04 17:52:24.613713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608153 2572 flags.go:64] FLAG: --cluster-dns="[]" Aug 04 17:52:24.613713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608157 2572 flags.go:64] FLAG: --cluster-domain="" Aug 04 17:52:24.613713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608160 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 04 17:52:24.613713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608163 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 04 17:52:24.613713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608166 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 04 17:52:24.613713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608169 2572 flags.go:64] FLAG: --container-log-max-files="5" Aug 04 17:52:24.613713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608174 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 04 17:52:24.613713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608178 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 04 17:52:24.613713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608181 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 04 17:52:24.613713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608184 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 04 17:52:24.613713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608187 2572 flags.go:64] FLAG: --contention-profiling="false" Aug 04 17:52:24.613713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608190 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 04 17:52:24.613713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608192 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 04 17:52:24.613713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608195 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608198 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608202 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608205 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608208 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608210 2572 flags.go:64] FLAG: --enable-load-reader="false" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608213 2572 flags.go:64] FLAG: --enable-server="true" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608216 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608223 2572 flags.go:64] FLAG: --event-burst="100" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608226 2572 flags.go:64] FLAG: --event-qps="50" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608228 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608231 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608234 2572 flags.go:64] FLAG: --eviction-hard="" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608238 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608241 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608243 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608246 2572 flags.go:64] FLAG: --eviction-soft="" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608249 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608252 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608255 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608258 2572 flags.go:64] FLAG: --experimental-mounter-path="" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608261 2572 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608263 2572 flags.go:64] FLAG: --fail-swap-on="true" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608266 2572 flags.go:64] FLAG: --feature-gates="" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608269 2572 flags.go:64] FLAG: --file-check-frequency="20s" Aug 04 17:52:24.614313 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608273 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 04 17:52:24.614947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608277 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 04 17:52:24.614947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608280 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 04 17:52:24.614947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608283 2572 flags.go:64] FLAG: --healthz-port="10248" Aug 04 17:52:24.614947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608286 2572 flags.go:64] FLAG: --help="false" Aug 04 17:52:24.614947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608289 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-137-173.ec2.internal" Aug 04 17:52:24.614947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608292 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 04 17:52:24.614947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608295 2572 flags.go:64] FLAG: --http-check-frequency="20s" Aug 04 17:52:24.614947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608298 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 04 17:52:24.614947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608301 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 04 17:52:24.614947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608305 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 04 17:52:24.614947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608307 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 04 17:52:24.614947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608310 2572 flags.go:64] FLAG: --image-service-endpoint="" Aug 04 17:52:24.614947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608313 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 04 17:52:24.614947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608316 2572 flags.go:64] FLAG: --kube-api-burst="100" Aug 04 17:52:24.614947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608318 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 04 17:52:24.614947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608321 2572 flags.go:64] FLAG: --kube-api-qps="50" Aug 04 17:52:24.614947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608334 2572 flags.go:64] FLAG: --kube-reserved="" Aug 04 17:52:24.614947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608338 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 04 17:52:24.614947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608341 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 04 17:52:24.614947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608344 2572 flags.go:64] FLAG: --kubelet-cgroups="" Aug 04 17:52:24.614947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608347 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 04 17:52:24.614947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608350 2572 flags.go:64] FLAG: --lock-file="" Aug 04 17:52:24.614947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608352 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 04 17:52:24.614947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608355 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 04 17:52:24.615527 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608358 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 04 17:52:24.615527 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608363 2572 flags.go:64] FLAG: --log-json-split-stream="false" Aug 04 17:52:24.615527 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608366 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 04 17:52:24.615527 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608369 2572 flags.go:64] FLAG: --log-text-split-stream="false" Aug 04 17:52:24.615527 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608372 2572 flags.go:64] FLAG: --logging-format="text" Aug 04 17:52:24.615527 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608375 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 04 17:52:24.615527 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608378 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 04 17:52:24.615527 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608382 2572 flags.go:64] FLAG: --manifest-url="" Aug 04 17:52:24.615527 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608386 2572 flags.go:64] FLAG: --manifest-url-header="" Aug 04 17:52:24.615527 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608390 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 04 17:52:24.615527 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608393 2572 flags.go:64] FLAG: --max-open-files="1000000" Aug 04 17:52:24.615527 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608397 2572 flags.go:64] FLAG: --max-pods="110" Aug 04 17:52:24.615527 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608400 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 04 17:52:24.615527 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608403 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 04 17:52:24.615527 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608406 2572 flags.go:64] FLAG: --memory-manager-policy="None" Aug 04 17:52:24.615527 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608409 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 04 17:52:24.615527 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608412 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 04 17:52:24.615527 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608415 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 04 17:52:24.615527 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608417 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 04 17:52:24.615527 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608424 2572 flags.go:64] FLAG: --node-status-max-images="50" Aug 04 17:52:24.615527 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608427 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 04 17:52:24.615527 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608430 2572 flags.go:64] FLAG: --oom-score-adj="-999" Aug 04 17:52:24.615527 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608433 2572 flags.go:64] FLAG: --pod-cidr="" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608435 2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608440 2572 flags.go:64] FLAG: --pod-manifest-path="" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608443 2572 flags.go:64] FLAG: --pod-max-pids="-1" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608446 2572 flags.go:64] FLAG: --pods-per-core="0" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608449 2572 flags.go:64] FLAG: --port="10250" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608452 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608454 2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f258d2e13d91abbf" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608457 2572 flags.go:64] FLAG: --qos-reserved="" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608460 2572 flags.go:64] FLAG: --read-only-port="10255" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608463 2572 flags.go:64] FLAG: --register-node="true" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608466 2572 flags.go:64] FLAG: --register-with-taints="" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608469 2572 flags.go:64] FLAG: --registry-burst="10" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608472 2572 flags.go:64] FLAG: --registry-qps="5" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608475 2572 flags.go:64] FLAG: --reserved-cpus="" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608478 2572 flags.go:64] FLAG: --reserved-memory="" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608481 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608484 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608489 2572 flags.go:64] FLAG: --rotate-certificates="false" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608492 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608494 2572 flags.go:64] FLAG: --runonce="false" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608497 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608500 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608503 2572 flags.go:64] FLAG: --seccomp-default="false" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608505 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 04 17:52:24.616091 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608508 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608510 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608513 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608516 2572 flags.go:64] FLAG: --storage-driver-password="root" Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608519 2572 flags.go:64] FLAG: --storage-driver-secure="false" Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608522 2572 flags.go:64] FLAG: --storage-driver-table="stats" Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608524 2572 flags.go:64] FLAG: --storage-driver-user="root" Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608527 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608530 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608533 2572 flags.go:64] FLAG: --system-cgroups="" Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608535 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608541 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608543 2572 flags.go:64] FLAG: --tls-cert-file="" Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608546 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608552 2572 flags.go:64] FLAG: --tls-min-version="" Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608555 2572 flags.go:64] FLAG: --tls-private-key-file="" Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608557 2572 flags.go:64] FLAG: --topology-manager-policy="none" Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608560 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608563 2572 flags.go:64] FLAG: --topology-manager-scope="container" Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608565 2572 flags.go:64] FLAG: --v="2" Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608569 2572 flags.go:64] FLAG: --version="false" Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608573 2572 flags.go:64] FLAG: --vmodule="" Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608576 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608579 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608714 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 17:52:24.616750 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608719 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 17:52:24.617339 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608723 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 17:52:24.617339 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608726 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 17:52:24.617339 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608728 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 17:52:24.617339 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608731 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 17:52:24.617339 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608735 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 17:52:24.617339 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608737 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 17:52:24.617339 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608740 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 17:52:24.617339 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608743 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 17:52:24.617339 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608746 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 17:52:24.617339 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608750 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 17:52:24.617339 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608753 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 17:52:24.617339 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608756 2572 feature_gate.go:328] unrecognized feature gate: Example Aug 04 17:52:24.617339 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608759 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 17:52:24.617339 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608763 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 17:52:24.617339 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608765 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 17:52:24.617339 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608768 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 17:52:24.617339 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608770 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 17:52:24.617339 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608773 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 17:52:24.617339 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608776 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 17:52:24.617339 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608778 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 17:52:24.617866 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608781 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 17:52:24.617866 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608785 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 17:52:24.617866 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608788 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 17:52:24.617866 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608791 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 17:52:24.617866 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608794 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 17:52:24.617866 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608796 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 17:52:24.617866 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608799 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 17:52:24.617866 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608801 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 17:52:24.617866 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608804 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 17:52:24.617866 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608807 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 17:52:24.617866 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608810 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 17:52:24.617866 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608813 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 17:52:24.617866 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608817 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 17:52:24.617866 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608820 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 17:52:24.617866 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608823 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 17:52:24.617866 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608826 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 17:52:24.617866 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608829 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 17:52:24.617866 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608831 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 17:52:24.617866 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608834 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 17:52:24.618314 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608836 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 17:52:24.618314 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608839 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 17:52:24.618314 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608842 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 17:52:24.618314 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608845 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 17:52:24.618314 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608847 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 17:52:24.618314 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608849 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 17:52:24.618314 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608852 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 17:52:24.618314 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608854 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 17:52:24.618314 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608857 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 17:52:24.618314 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608859 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 17:52:24.618314 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608862 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 17:52:24.618314 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608864 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 17:52:24.618314 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608866 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 17:52:24.618314 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608869 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 17:52:24.618314 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608871 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 17:52:24.618314 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608874 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 17:52:24.618314 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608878 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 17:52:24.618314 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608880 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 17:52:24.618314 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608883 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 17:52:24.618314 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608885 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 17:52:24.618814 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608888 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 17:52:24.618814 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608890 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 17:52:24.618814 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608893 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 17:52:24.618814 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608895 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 17:52:24.618814 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608899 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 17:52:24.618814 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608901 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 17:52:24.618814 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608904 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 17:52:24.618814 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608906 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 17:52:24.618814 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608909 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 17:52:24.618814 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608911 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 17:52:24.618814 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608913 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 17:52:24.618814 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608916 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 17:52:24.618814 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608918 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 17:52:24.618814 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608921 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 17:52:24.618814 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608923 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 17:52:24.618814 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608925 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 17:52:24.618814 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608928 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 17:52:24.618814 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608931 2572 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 17:52:24.618814 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608933 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 17:52:24.618814 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608936 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 17:52:24.619296 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608938 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 17:52:24.619296 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608940 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 17:52:24.619296 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608943 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 17:52:24.619296 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608945 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 17:52:24.619296 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608947 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 17:52:24.619296 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608950 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 17:52:24.619296 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608952 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 17:52:24.619296 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608955 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 17:52:24.619296 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608958 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 17:52:24.619296 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608961 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 17:52:24.619296 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608963 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 17:52:24.619296 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608966 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 17:52:24.619296 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608968 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 17:52:24.619296 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608970 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 17:52:24.619296 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608973 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 17:52:24.619296 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608976 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 17:52:24.619296 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608980 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 17:52:24.619296 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.608983 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 17:52:24.619757 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.608991 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 17:52:24.619757 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.615989 2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 04 17:52:24.619757 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.616006 2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 04 17:52:24.619757 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.616024 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 04 17:52:24.619757 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.616030 2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 04 17:52:24.619757 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616089 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 17:52:24.619757 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616094 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 17:52:24.619757 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616099 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 17:52:24.619757 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616103 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 17:52:24.619757 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616107 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 17:52:24.619757 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616110 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 17:52:24.619757 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616113 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 17:52:24.619757 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616115 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 17:52:24.619757 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616118 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 17:52:24.620116 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616122 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 17:52:24.620116 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616125 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 17:52:24.620116 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616127 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 17:52:24.620116 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616130 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 17:52:24.620116 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616133 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 17:52:24.620116 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616135 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 17:52:24.620116 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616138 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 17:52:24.620116 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616140 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 17:52:24.620116 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616142 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 17:52:24.620116 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616145 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 17:52:24.620116 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616147 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 17:52:24.620116 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616150 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 17:52:24.620116 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616152 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 17:52:24.620116 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616155 2572 feature_gate.go:328] unrecognized feature gate: Example Aug 04 17:52:24.620116 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616157 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 17:52:24.620116 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616160 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 17:52:24.620116 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616163 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 17:52:24.620116 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616167 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 17:52:24.620116 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616170 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 17:52:24.620573 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616172 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 17:52:24.620573 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616176 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 17:52:24.620573 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616179 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 17:52:24.620573 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616181 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 17:52:24.620573 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616184 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 17:52:24.620573 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616187 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 17:52:24.620573 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616189 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 17:52:24.620573 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616192 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 17:52:24.620573 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616194 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 17:52:24.620573 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616197 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 17:52:24.620573 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616199 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 17:52:24.620573 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616201 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 17:52:24.620573 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616204 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 17:52:24.620573 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616207 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 17:52:24.620573 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616210 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 17:52:24.620573 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616213 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 17:52:24.620573 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616216 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 17:52:24.620573 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616218 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 17:52:24.620573 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616221 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 17:52:24.621114 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616223 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 17:52:24.621114 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616226 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 17:52:24.621114 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616228 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 17:52:24.621114 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616231 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 17:52:24.621114 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616233 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 17:52:24.621114 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616236 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 17:52:24.621114 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616238 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 17:52:24.621114 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616241 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 17:52:24.621114 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616244 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 17:52:24.621114 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616246 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 17:52:24.621114 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616249 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 17:52:24.621114 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616251 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 17:52:24.621114 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616253 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 17:52:24.621114 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616256 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 17:52:24.621114 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616259 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 17:52:24.621114 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616262 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 17:52:24.621114 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616265 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 17:52:24.621114 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616267 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 17:52:24.621114 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616269 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 17:52:24.621114 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616272 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 17:52:24.621598 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616274 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 17:52:24.621598 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616276 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 17:52:24.621598 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616279 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 17:52:24.621598 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616281 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 17:52:24.621598 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616283 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 17:52:24.621598 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616286 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 17:52:24.621598 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616289 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 17:52:24.621598 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616291 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 17:52:24.621598 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616294 2572 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 17:52:24.621598 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616296 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 17:52:24.621598 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616298 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 17:52:24.621598 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616301 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 17:52:24.621598 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616303 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 17:52:24.621598 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616306 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 17:52:24.621598 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616308 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 17:52:24.621598 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616310 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 17:52:24.621598 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616313 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 17:52:24.621598 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616316 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 17:52:24.621598 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616319 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 17:52:24.621598 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616321 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 17:52:24.622084 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616324 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 17:52:24.622084 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616326 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 17:52:24.622084 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616329 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 17:52:24.622084 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616331 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 17:52:24.622084 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616334 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 17:52:24.622084 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616336 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 17:52:24.622084 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616339 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 17:52:24.622084 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616341 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 17:52:24.622084 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616352 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 17:52:24.622084 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616355 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 17:52:24.622084 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616357 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 17:52:24.622084 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616359 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 17:52:24.622084 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.616365 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 17:52:24.622084 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616491 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 17:52:24.622084 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616495 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 17:52:24.622463 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616498 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 17:52:24.622463 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616500 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 17:52:24.622463 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616503 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 17:52:24.622463 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616506 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 17:52:24.622463 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616509 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 17:52:24.622463 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616512 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 17:52:24.622463 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616515 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 17:52:24.622463 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616517 2572 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 17:52:24.622463 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616520 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 17:52:24.622463 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616523 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 17:52:24.622463 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616525 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 17:52:24.622463 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616528 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 17:52:24.622463 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616530 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 17:52:24.622463 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616533 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 17:52:24.622463 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616535 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 17:52:24.622463 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616539 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 17:52:24.622463 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616543 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 17:52:24.622463 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616546 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 17:52:24.622463 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616549 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 17:52:24.622463 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616551 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 17:52:24.622962 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616554 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 17:52:24.622962 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616557 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 17:52:24.622962 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616559 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 17:52:24.622962 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616562 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 17:52:24.622962 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616564 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 17:52:24.622962 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616567 2572 feature_gate.go:328] unrecognized feature gate: Example Aug 04 17:52:24.622962 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616575 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 17:52:24.622962 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616578 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 17:52:24.622962 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616580 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 17:52:24.622962 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616582 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 17:52:24.622962 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616585 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 17:52:24.622962 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616587 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 17:52:24.622962 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616590 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 17:52:24.622962 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616592 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 17:52:24.622962 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616594 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 17:52:24.622962 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616597 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 17:52:24.622962 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616599 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 17:52:24.622962 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616601 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 17:52:24.622962 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616604 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 17:52:24.622962 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616607 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 17:52:24.623438 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616609 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 17:52:24.623438 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616612 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 17:52:24.623438 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616614 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 17:52:24.623438 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616617 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 17:52:24.623438 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616619 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 17:52:24.623438 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616621 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 17:52:24.623438 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616624 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 17:52:24.623438 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616627 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 17:52:24.623438 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616629 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 17:52:24.623438 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616631 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 17:52:24.623438 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616634 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 17:52:24.623438 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616636 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 17:52:24.623438 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616639 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 17:52:24.623438 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616641 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 17:52:24.623438 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616644 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 17:52:24.623438 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616646 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 17:52:24.623438 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616649 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 17:52:24.623438 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616651 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 17:52:24.623438 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616654 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 17:52:24.623916 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616679 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 17:52:24.623916 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616683 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 17:52:24.623916 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616686 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 17:52:24.623916 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616688 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 17:52:24.623916 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616691 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 17:52:24.623916 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616694 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 17:52:24.623916 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616696 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 17:52:24.623916 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616699 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 17:52:24.623916 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616701 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 17:52:24.623916 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616704 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 17:52:24.623916 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616706 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 17:52:24.623916 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616709 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 17:52:24.623916 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616711 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 17:52:24.623916 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616713 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 17:52:24.623916 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616716 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 17:52:24.623916 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616718 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 17:52:24.623916 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616721 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 17:52:24.623916 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616723 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 17:52:24.623916 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616725 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 17:52:24.623916 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616728 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 17:52:24.624379 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616730 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 17:52:24.624379 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616733 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 17:52:24.624379 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616735 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 17:52:24.624379 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616737 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 17:52:24.624379 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616740 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 17:52:24.624379 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616744 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 17:52:24.624379 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616747 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 17:52:24.624379 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616750 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 17:52:24.624379 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616752 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 17:52:24.624379 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616755 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 17:52:24.624379 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616758 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 17:52:24.624379 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616761 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 17:52:24.624379 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616764 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 17:52:24.624379 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616773 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 17:52:24.624379 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616776 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 17:52:24.624379 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616780 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 17:52:24.624379 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616782 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 17:52:24.624379 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.616786 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 17:52:24.624878 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.616791 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 17:52:24.624878 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.616895 2572 server.go:962] "Client rotation is on, will bootstrap in background" Aug 04 17:52:24.624878 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.619772 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 04 17:52:24.624878 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.621267 2572 server.go:1019] "Starting client certificate rotation" Aug 04 17:52:24.624878 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.621407 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 17:52:24.624878 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.621445 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 17:52:24.624878 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.623979 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 17:52:24.625541 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.625519 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 17:52:24.633940 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.633920 2572 log.go:25] "Validated CRI v1 runtime API" Aug 04 17:52:24.636054 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.636039 2572 log.go:25] "Validated CRI v1 image API" Aug 04 17:52:24.637823 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.637804 2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 04 17:52:24.643699 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.643679 2572 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8a3be0a0-da98-4e42-a944-8cff02798a94:/dev/nvme0n1p4 a73c0d0d-f2ad-4f60-94ea-da5d5b0dd4c1:/dev/nvme0n1p3] Aug 04 17:52:24.643760 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.643698 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 04 17:52:24.648707 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.648588 2572 manager.go:216] Machine: {Timestamp:2026-08-04 17:52:24.647234086 +0000 UTC m=+0.337925128 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3145230 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec298e0c86ed0d5c576ae35f197ebeb9 SystemUUID:ec298e0c-86ed-0d5c-576a-e35f197ebeb9 BootID:b1c58239-cc44-4156-8e2a-62f42737af8d Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f5:a2:f4:fd:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f5:a2:f4:fd:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:17:79:68:54:0e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 04 17:52:24.648707 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.648703 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 04 17:52:24.648809 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.648774 2572 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 04 17:52:24.652634 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.652605 2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 04 17:52:24.652791 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.652640 2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-173.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 04 17:52:24.654084 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.654073 2572 topology_manager.go:138] "Creating topology manager with none policy" Aug 04 17:52:24.654117 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.654088 2572 container_manager_linux.go:309] "Creating device plugin manager" Aug 04 17:52:24.654117 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.654103 2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 17:52:24.654172 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.654124 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 17:52:24.655056 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.655046 2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 04 17:52:24.656413 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.656389 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 17:52:24.658337 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.658324 2572 state_mem.go:36] "Initialized new in-memory state store" Aug 04 17:52:24.658453 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.658444 2572 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 04 17:52:24.658492 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.658477 2572 kubelet.go:486] "Attempting to sync node with API server" Aug 04 17:52:24.658534 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.658493 2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 04 17:52:24.658534 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.658514 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 04 17:52:24.658534 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.658523 2572 kubelet.go:398] "Adding apiserver pod source" Aug 04 17:52:24.658534 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.658532 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 04 17:52:24.658872 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.658860 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 17:52:24.658924 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.658878 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 17:52:24.663234 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.663217 2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 04 17:52:24.664595 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.664582 2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 04 17:52:24.664645 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.664612 2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 04 17:52:24.665927 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.665914 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 04 17:52:24.665970 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.665935 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 04 17:52:24.665970 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.665945 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 04 17:52:24.665970 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.665954 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 04 17:52:24.665970 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.665966 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 04 17:52:24.666078 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.665973 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 04 17:52:24.666078 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.665985 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 04 17:52:24.666078 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.665992 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 04 17:52:24.666078 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.666003 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 04 17:52:24.666078 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.666011 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 04 17:52:24.666078 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.666023 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 04 17:52:24.666078 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.666032 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 04 17:52:24.667387 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.667374 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 04 17:52:24.667432 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.667388 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 04 17:52:24.668681 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:24.668641 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 04 17:52:24.668767 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:24.668653 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-173.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 04 17:52:24.670806 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.670791 2572 server.go:1268] "Started kubelet" Aug 04 17:52:24.671134 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.671073 2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 04 17:52:24.671225 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.671146 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 04 17:52:24.671225 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.671212 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 04 17:52:24.672131 ip-10-0-137-173 systemd[1]: Started Kubernetes Kubelet. Aug 04 17:52:24.672874 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.672807 2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 04 17:52:24.675089 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.675073 2572 server.go:310] "Adding debug handlers to kubelet server" Aug 04 17:52:24.678726 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:24.678707 2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 04 17:52:24.680339 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.680323 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-173.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 04 17:52:24.680424 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.680409 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wqssx" Aug 04 17:52:24.681219 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:24.680243 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-173.ec2.internal.18c8ab9d609177cc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-173.ec2.internal,UID:ip-10-0-137-173.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-173.ec2.internal,},FirstTimestamp:2026-08-04 17:52:24.670762956 +0000 UTC m=+0.361454054,LastTimestamp:2026-08-04 17:52:24.670762956 +0000 UTC m=+0.361454054,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-173.ec2.internal,}" Aug 04 17:52:24.681819 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.681804 2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 04 17:52:24.681910 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.681850 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 04 17:52:24.681961 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:24.681916 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-173.ec2.internal\" not found" Aug 04 17:52:24.682562 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.682540 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 04 17:52:24.682639 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.682552 2572 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 04 17:52:24.682732 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.682655 2572 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 04 17:52:24.682775 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.682762 2572 factory.go:55] Registering systemd factory Aug 04 17:52:24.682812 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.682778 2572 factory.go:223] Registration of the systemd container factory successfully Aug 04 17:52:24.682887 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.682876 2572 reconstruct.go:97] "Volume reconstruction finished" Aug 04 17:52:24.682937 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.682887 2572 reconciler.go:29] "Reconciler: start to sync state" Aug 04 17:52:24.683080 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.683062 2572 factory.go:153] Registering CRI-O factory Aug 04 17:52:24.683135 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.683086 2572 factory.go:223] Registration of the crio container factory successfully Aug 04 17:52:24.683185 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.683145 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 04 17:52:24.683185 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.683170 2572 factory.go:103] Registering Raw factory Aug 04 17:52:24.683185 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.683185 2572 manager.go:1196] Started watching for new ooms in manager Aug 04 17:52:24.683610 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.683598 2572 manager.go:349] Starting recovery of all containers Aug 04 17:52:24.688263 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:24.688236 2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-173.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 04 17:52:24.688467 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:24.688443 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 04 17:52:24.690904 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.690729 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 04 17:52:24.690996 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.690776 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wqssx" Aug 04 17:52:24.692284 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:24.692255 2572 helpers.go:245] readString: Failed to read "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service/pids.max": read /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service/pids.max: no such device Aug 04 17:52:24.695114 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.695092 2572 manager.go:354] Recovery completed Aug 04 17:52:24.700085 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.700068 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 17:52:24.701748 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.701733 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientMemory" Aug 04 17:52:24.701813 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.701762 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 17:52:24.701813 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.701775 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientPID" Aug 04 17:52:24.702326 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.702309 2572 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 04 17:52:24.702326 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.702323 2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 04 17:52:24.702428 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.702343 2572 state_mem.go:36] "Initialized new in-memory state store" Aug 04 17:52:24.702428 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.702345 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 17:52:24.703136 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.703113 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientMemory" Aug 04 17:52:24.703182 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.703138 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 17:52:24.703182 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.703157 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientPID" Aug 04 17:52:24.703326 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.703314 2572 policy_none.go:49] "None policy: Start" Aug 04 17:52:24.703384 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.703333 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 04 17:52:24.703384 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.703347 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 04 17:52:24.704281 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:24.704205 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-173.ec2.internal.18c8ab9d626a4268 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-173.ec2.internal,UID:ip-10-0-137-173.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-173.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-173.ec2.internal,},FirstTimestamp:2026-08-04 17:52:24.701747816 +0000 UTC m=+0.392438858,LastTimestamp:2026-08-04 17:52:24.701747816 +0000 UTC m=+0.392438858,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-173.ec2.internal,}" Aug 04 17:52:24.704370 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.704298 2572 policy_none.go:47] "Start" Aug 04 17:52:24.743295 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.743267 2572 manager.go:337] "Starting Device Plugin manager" Aug 04 17:52:24.743374 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:24.743303 2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 04 17:52:24.743374 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.743317 2572 server.go:87] "Starting device plugin registration server" Aug 04 17:52:24.743617 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.743604 2572 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 04 17:52:24.743685 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.743625 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 04 17:52:24.743770 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.743734 2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 04 17:52:24.743874 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.743834 2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 04 17:52:24.743874 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.743849 2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 04 17:52:24.744358 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:24.744337 2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 04 17:52:24.744437 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:24.744394 2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-173.ec2.internal\" not found" Aug 04 17:52:24.815445 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.815410 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 04 17:52:24.816599 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.815457 2572 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 04 17:52:24.816599 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.815485 2572 kubelet.go:2448] "Starting kubelet main sync loop" Aug 04 17:52:24.816599 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:24.815528 2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 04 17:52:24.818069 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.818051 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 17:52:24.844561 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.844532 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 17:52:24.845508 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.845492 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientMemory" Aug 04 17:52:24.845603 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.845525 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 17:52:24.845603 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.845541 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientPID" Aug 04 17:52:24.845603 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.845565 2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-173.ec2.internal" Aug 04 17:52:24.855488 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.855430 2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-173.ec2.internal" Aug 04 17:52:24.855488 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:24.855454 2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-173.ec2.internal\": node \"ip-10-0-137-173.ec2.internal\" not found" Aug 04 17:52:24.872570 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:24.872549 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-173.ec2.internal\" not found" Aug 04 17:52:24.915905 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.915880 2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-173.ec2.internal"] Aug 04 17:52:24.916018 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.915943 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 17:52:24.916879 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.916863 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientMemory" Aug 04 17:52:24.916933 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.916894 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 17:52:24.916933 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.916904 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientPID" Aug 04 17:52:24.918200 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.918188 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 17:52:24.918315 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.918302 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" Aug 04 17:52:24.918359 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.918351 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 17:52:24.918889 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.918876 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientMemory" Aug 04 17:52:24.918938 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.918901 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientMemory" Aug 04 17:52:24.918977 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.918939 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 17:52:24.918977 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.918949 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientPID" Aug 04 17:52:24.918977 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.918906 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 17:52:24.919070 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.919000 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientPID" Aug 04 17:52:24.920071 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.920056 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-173.ec2.internal" Aug 04 17:52:24.920131 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.920079 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 17:52:24.920736 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.920723 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientMemory" Aug 04 17:52:24.920795 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.920749 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 17:52:24.920795 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.920763 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientPID" Aug 04 17:52:24.938196 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:24.938173 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-173.ec2.internal\" not found" node="ip-10-0-137-173.ec2.internal" Aug 04 17:52:24.941473 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:24.941459 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-173.ec2.internal\" not found" node="ip-10-0-137-173.ec2.internal" Aug 04 17:52:24.972876 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:24.972853 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-173.ec2.internal\" not found" Aug 04 17:52:24.985980 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.985951 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ea37b25d2fb147b8b95f79511393014-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal\" (UID: \"9ea37b25d2fb147b8b95f79511393014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" Aug 04 17:52:24.986053 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.985985 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ea37b25d2fb147b8b95f79511393014-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal\" (UID: \"9ea37b25d2fb147b8b95f79511393014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" Aug 04 17:52:24.986053 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:24.986003 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37a95ca805868321653a87ed06a2c559-config\") pod \"kube-apiserver-proxy-ip-10-0-137-173.ec2.internal\" (UID: \"37a95ca805868321653a87ed06a2c559\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-173.ec2.internal" Aug 04 17:52:25.073223 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:25.073192 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-173.ec2.internal\" not found" Aug 04 17:52:25.086642 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:25.086620 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ea37b25d2fb147b8b95f79511393014-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal\" (UID: \"9ea37b25d2fb147b8b95f79511393014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" Aug 04 17:52:25.086713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:25.086647 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ea37b25d2fb147b8b95f79511393014-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal\" (UID: \"9ea37b25d2fb147b8b95f79511393014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" Aug 04 17:52:25.086713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:25.086675 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37a95ca805868321653a87ed06a2c559-config\") pod \"kube-apiserver-proxy-ip-10-0-137-173.ec2.internal\" (UID: \"37a95ca805868321653a87ed06a2c559\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-173.ec2.internal" Aug 04 17:52:25.086779 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:25.086723 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37a95ca805868321653a87ed06a2c559-config\") pod \"kube-apiserver-proxy-ip-10-0-137-173.ec2.internal\" (UID: \"37a95ca805868321653a87ed06a2c559\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-173.ec2.internal" Aug 04 17:52:25.086779 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:25.086735 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ea37b25d2fb147b8b95f79511393014-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal\" (UID: \"9ea37b25d2fb147b8b95f79511393014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" Aug 04 17:52:25.086837 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:25.086785 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ea37b25d2fb147b8b95f79511393014-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal\" (UID: \"9ea37b25d2fb147b8b95f79511393014\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" Aug 04 17:52:25.174121 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:25.174041 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-173.ec2.internal\" not found" Aug 04 17:52:25.241632 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:25.241596 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" Aug 04 17:52:25.244874 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:25.244855 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-173.ec2.internal" Aug 04 17:52:25.274522 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:25.274481 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-173.ec2.internal\" not found" Aug 04 17:52:25.375070 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:25.375032 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-173.ec2.internal\" not found" Aug 04 17:52:25.475326 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:25.475265 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-173.ec2.internal\" not found" Aug 04 17:52:25.581907 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:25.581870 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-173.ec2.internal\" not found" Aug 04 17:52:25.619209 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:25.619189 2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 17:52:25.620762 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:25.620746 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 04 17:52:25.620882 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:25.620854 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 04 17:52:25.682054 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:25.682030 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-173.ec2.internal\" not found" Aug 04 17:52:25.695201 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:25.695173 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 17:47:25 +0000 UTC" deadline="2028-02-26 03:20:35.909281192 +0000 UTC" Aug 04 17:52:25.695201 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:25.695196 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13689h28m10.214088033s" Aug 04 17:52:25.782913 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:25.782861 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-173.ec2.internal\" not found" Aug 04 17:52:25.883949 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:25.883912 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-173.ec2.internal\" not found" Aug 04 17:52:25.910309 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:25.910289 2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 17:52:25.944077 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:25.944055 2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 17:52:25.947656 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:25.947621 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea37b25d2fb147b8b95f79511393014.slice/crio-57bf9de02cb3c1a278e51d1366415f2e9ee46b3a71c00da82743d2a99b7c0be9 WatchSource:0}: Error finding container 57bf9de02cb3c1a278e51d1366415f2e9ee46b3a71c00da82743d2a99b7c0be9: Status 404 returned error can't find the container with id 57bf9de02cb3c1a278e51d1366415f2e9ee46b3a71c00da82743d2a99b7c0be9 Aug 04 17:52:25.948284 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:25.948260 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a95ca805868321653a87ed06a2c559.slice/crio-82e1722474992a308aa100e4e6aa333c2e52d4861093c74522f9873d885d7539 WatchSource:0}: Error finding container 82e1722474992a308aa100e4e6aa333c2e52d4861093c74522f9873d885d7539: Status 404 returned error can't find the container with id 82e1722474992a308aa100e4e6aa333c2e52d4861093c74522f9873d885d7539 Aug 04 17:52:25.952750 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:25.952725 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 17:52:25.982440 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:25.982411 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" Aug 04 17:52:26.001545 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.001525 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 17:52:26.002509 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.002496 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-173.ec2.internal" Aug 04 17:52:26.013727 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.013704 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 17:52:26.660468 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.660437 2572 apiserver.go:52] "Watching apiserver" Aug 04 17:52:26.671448 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.671254 2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 04 17:52:26.672612 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.672581 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal","openshift-network-diagnostics/network-check-target-9z2cq","openshift-network-operator/iptables-alerter-66v2k","kube-system/kube-apiserver-proxy-ip-10-0-137-173.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll","openshift-image-registry/node-ca-6ns66","openshift-multus/multus-additional-cni-plugins-t2nnw","openshift-multus/multus-qqgm5","openshift-multus/network-metrics-daemon-mpgkn","openshift-ovn-kubernetes/ovnkube-node-2lkw4","kube-system/global-pull-secret-syncer-xbhvg","kube-system/konnectivity-agent-64c4r","openshift-cluster-node-tuning-operator/tuned-dt2dw","openshift-dns/node-resolver-4dgxz"] Aug 04 17:52:26.674567 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.674533 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.675718 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.675688 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.677217 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.677198 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-66v2k" Aug 04 17:52:26.677658 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.677638 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 04 17:52:26.677794 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.677775 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 04 17:52:26.678493 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.677952 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 04 17:52:26.678493 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.677978 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fh62v\"" Aug 04 17:52:26.678493 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.677979 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 04 17:52:26.678493 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.678140 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 04 17:52:26.678493 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.678167 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 04 17:52:26.678493 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.678249 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 04 17:52:26.678493 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.678291 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 04 17:52:26.678493 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.678350 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nbcb8\"" Aug 04 17:52:26.679942 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.679802 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 04 17:52:26.679942 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.679822 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-82cwc\"" Aug 04 17:52:26.680154 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.680135 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4dgxz" Aug 04 17:52:26.680922 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.680900 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 04 17:52:26.681016 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.680945 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 04 17:52:26.681507 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.681486 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" Aug 04 17:52:26.682987 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.682034 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 04 17:52:26.684048 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.684013 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 04 17:52:26.684488 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.684472 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 04 17:52:26.685430 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.685411 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6ns66" Aug 04 17:52:26.686993 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.686973 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:26.687120 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:26.687075 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:52:26.688678 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.688641 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.690094 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.690072 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:26.690193 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:26.690148 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:52:26.692228 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.692211 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.693597 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.693576 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:26.693802 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:26.693779 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:52:26.695380 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.695363 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-64c4r" Aug 04 17:52:26.697230 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.695577 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/610b5dc3-b664-4744-b311-85bbd40575b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-57kll\" (UID: \"610b5dc3-b664-4744-b311-85bbd40575b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" Aug 04 17:52:26.697230 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.695611 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/610b5dc3-b664-4744-b311-85bbd40575b4-sys-fs\") pod \"aws-ebs-csi-driver-node-57kll\" (UID: \"610b5dc3-b664-4744-b311-85bbd40575b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" Aug 04 17:52:26.697230 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.695638 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d005c33-5c1e-4ec8-990d-ee4d27743422-serviceca\") pod \"node-ca-6ns66\" (UID: \"8d005c33-5c1e-4ec8-990d-ee4d27743422\") " pod="openshift-image-registry/node-ca-6ns66" Aug 04 17:52:26.697230 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.695664 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-etc-openvswitch\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.697230 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.695689 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-run-ovn\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.697230 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.695742 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-sys\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.697230 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.695777 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s74b\" (UniqueName: \"kubernetes.io/projected/4fbeee23-8c17-49ea-b198-0b4d1d79a517-kube-api-access-8s74b\") pod \"iptables-alerter-66v2k\" (UID: \"4fbeee23-8c17-49ea-b198-0b4d1d79a517\") " pod="openshift-network-operator/iptables-alerter-66v2k" Aug 04 17:52:26.697230 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.695802 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/610b5dc3-b664-4744-b311-85bbd40575b4-device-dir\") pod \"aws-ebs-csi-driver-node-57kll\" (UID: \"610b5dc3-b664-4744-b311-85bbd40575b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" Aug 04 17:52:26.697230 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.695826 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-node-log\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.697230 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.695851 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-host-cni-bin\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.697230 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.695875 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wg98\" (UniqueName: \"kubernetes.io/projected/c11d0415-39ac-4682-81d2-130d039fae98-kube-api-access-2wg98\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.697230 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.695911 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-etc-kubernetes\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.697230 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.695935 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-etc-sysctl-conf\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.697230 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.695959 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-lib-modules\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.697230 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.695998 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.697230 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696024 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c11d0415-39ac-4682-81d2-130d039fae98-env-overrides\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.697883 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696050 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-etc-sysconfig\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.697883 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696075 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-etc-systemd\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.697883 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696097 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-run\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.697883 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696119 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3616edbe-d877-4980-b66a-e96223d78562-etc-tuned\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.697883 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696143 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qttzb\" (UniqueName: \"kubernetes.io/projected/610b5dc3-b664-4744-b311-85bbd40575b4-kube-api-access-qttzb\") pod \"aws-ebs-csi-driver-node-57kll\" (UID: \"610b5dc3-b664-4744-b311-85bbd40575b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" Aug 04 17:52:26.697883 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696166 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-host-kubelet\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.697883 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696201 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-host-cni-netd\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.697883 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696223 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-etc-modprobe-d\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.697883 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696246 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-host\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.697883 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696267 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4fbeee23-8c17-49ea-b198-0b4d1d79a517-iptables-alerter-script\") pod \"iptables-alerter-66v2k\" (UID: \"4fbeee23-8c17-49ea-b198-0b4d1d79a517\") " pod="openshift-network-operator/iptables-alerter-66v2k" Aug 04 17:52:26.697883 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696300 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g74lb\" (UniqueName: \"kubernetes.io/projected/c600bdf9-3cef-457a-bb0b-704c5856c077-kube-api-access-g74lb\") pod \"node-resolver-4dgxz\" (UID: \"c600bdf9-3cef-457a-bb0b-704c5856c077\") " pod="openshift-dns/node-resolver-4dgxz" Aug 04 17:52:26.697883 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696339 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-var-lib-openvswitch\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.697883 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696363 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c600bdf9-3cef-457a-bb0b-704c5856c077-hosts-file\") pod \"node-resolver-4dgxz\" (UID: \"c600bdf9-3cef-457a-bb0b-704c5856c077\") " pod="openshift-dns/node-resolver-4dgxz" Aug 04 17:52:26.697883 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696388 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c600bdf9-3cef-457a-bb0b-704c5856c077-tmp-dir\") pod \"node-resolver-4dgxz\" (UID: \"c600bdf9-3cef-457a-bb0b-704c5856c077\") " pod="openshift-dns/node-resolver-4dgxz" Aug 04 17:52:26.697883 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696415 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/610b5dc3-b664-4744-b311-85bbd40575b4-socket-dir\") pod \"aws-ebs-csi-driver-node-57kll\" (UID: \"610b5dc3-b664-4744-b311-85bbd40575b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" Aug 04 17:52:26.697883 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696443 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kb4vj\"" Aug 04 17:52:26.697883 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696446 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/610b5dc3-b664-4744-b311-85bbd40575b4-registration-dir\") pod \"aws-ebs-csi-driver-node-57kll\" (UID: \"610b5dc3-b664-4744-b311-85bbd40575b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" Aug 04 17:52:26.698715 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696483 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-run-openvswitch\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.698715 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696507 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c11d0415-39ac-4682-81d2-130d039fae98-ovn-node-metrics-cert\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.698715 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696542 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-etc-sysctl-d\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.698715 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696567 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzh56\" (UniqueName: \"kubernetes.io/projected/3616edbe-d877-4980-b66a-e96223d78562-kube-api-access-gzh56\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.698715 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696592 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fbeee23-8c17-49ea-b198-0b4d1d79a517-host-slash\") pod \"iptables-alerter-66v2k\" (UID: \"4fbeee23-8c17-49ea-b198-0b4d1d79a517\") " pod="openshift-network-operator/iptables-alerter-66v2k" Aug 04 17:52:26.698715 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696617 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plxpt\" (UniqueName: \"kubernetes.io/projected/8d005c33-5c1e-4ec8-990d-ee4d27743422-kube-api-access-plxpt\") pod \"node-ca-6ns66\" (UID: \"8d005c33-5c1e-4ec8-990d-ee4d27743422\") " pod="openshift-image-registry/node-ca-6ns66" Aug 04 17:52:26.698715 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696646 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-host-slash\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.698715 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696675 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-host-run-netns\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.698715 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.696723 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-run-systemd\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.698715 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.697149 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z2l9x\"" Aug 04 17:52:26.698715 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.697411 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 04 17:52:26.698715 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.697618 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qbnb6\"" Aug 04 17:52:26.698715 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.698678 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d005c33-5c1e-4ec8-990d-ee4d27743422-host\") pod \"node-ca-6ns66\" (UID: \"8d005c33-5c1e-4ec8-990d-ee4d27743422\") " pod="openshift-image-registry/node-ca-6ns66" Aug 04 17:52:26.699374 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.698757 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-systemd-units\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.699374 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.698811 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-log-socket\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.699374 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.698840 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-host-run-ovn-kubernetes\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.699374 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.698875 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c11d0415-39ac-4682-81d2-130d039fae98-ovnkube-config\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.699374 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.698908 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c11d0415-39ac-4682-81d2-130d039fae98-ovnkube-script-lib\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.699374 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.698942 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-var-lib-kubelet\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.699374 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.698975 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3616edbe-d877-4980-b66a-e96223d78562-tmp\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.699374 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.699002 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/610b5dc3-b664-4744-b311-85bbd40575b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-57kll\" (UID: \"610b5dc3-b664-4744-b311-85bbd40575b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" Aug 04 17:52:26.699804 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.699482 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-w96m9\"" Aug 04 17:52:26.701205 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.701185 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 04 17:52:26.701500 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.701486 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 04 17:52:26.701687 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.701673 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 04 17:52:26.702141 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.702121 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 04 17:52:26.702298 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.702285 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 04 17:52:26.702450 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.702437 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 04 17:52:26.702598 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.702585 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 04 17:52:26.702971 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.702950 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 04 17:52:26.703316 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.703297 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 04 17:52:26.703389 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.703316 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 04 17:52:26.703825 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.703807 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 04 17:52:26.703916 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.703834 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fmq7d\"" Aug 04 17:52:26.704088 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.704060 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cxk77\"" Aug 04 17:52:26.704607 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.704589 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 04 17:52:26.706970 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.706952 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 04 17:52:26.713735 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.713709 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 17:52:26.722972 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.722948 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tldpc" Aug 04 17:52:26.732724 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.732705 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tldpc" Aug 04 17:52:26.783682 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.783645 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 04 17:52:26.799589 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.799554 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s74b\" (UniqueName: \"kubernetes.io/projected/4fbeee23-8c17-49ea-b198-0b4d1d79a517-kube-api-access-8s74b\") pod \"iptables-alerter-66v2k\" (UID: \"4fbeee23-8c17-49ea-b198-0b4d1d79a517\") " pod="openshift-network-operator/iptables-alerter-66v2k" Aug 04 17:52:26.799589 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.799595 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-host-cni-bin\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.799856 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.799633 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a1dbfaf6-9a2f-4146-807c-418181c922c1-os-release\") pod \"multus-additional-cni-plugins-t2nnw\" (UID: \"a1dbfaf6-9a2f-4146-807c-418181c922c1\") " pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.799856 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.799659 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-multus-socket-dir-parent\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.799856 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.799685 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-etc-kubernetes\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.799856 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.799709 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-lib-modules\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.799856 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.799755 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-host-cni-bin\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.799856 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.799758 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.799856 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.799809 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c11d0415-39ac-4682-81d2-130d039fae98-env-overrides\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.799856 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.799837 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-etc-kubernetes\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.799856 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.799845 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.800306 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.799880 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1dbfaf6-9a2f-4146-807c-418181c922c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t2nnw\" (UID: \"a1dbfaf6-9a2f-4146-807c-418181c922c1\") " pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.800306 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800085 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-lib-modules\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.800306 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800113 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-multus-cni-dir\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.800306 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800130 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-os-release\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.800306 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800162 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-479kg\" (UniqueName: \"kubernetes.io/projected/4bb95b54-5442-43b6-befa-8ecf7b8f4781-kube-api-access-479kg\") pod \"network-check-target-9z2cq\" (UID: \"4bb95b54-5442-43b6-befa-8ecf7b8f4781\") " pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:26.800306 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800183 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qttzb\" (UniqueName: \"kubernetes.io/projected/610b5dc3-b664-4744-b311-85bbd40575b4-kube-api-access-qttzb\") pod \"aws-ebs-csi-driver-node-57kll\" (UID: \"610b5dc3-b664-4744-b311-85bbd40575b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" Aug 04 17:52:26.800306 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800235 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-host-var-lib-cni-multus\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.800306 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800274 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-etc-modprobe-d\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.800306 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800299 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4fbeee23-8c17-49ea-b198-0b4d1d79a517-iptables-alerter-script\") pod \"iptables-alerter-66v2k\" (UID: \"4fbeee23-8c17-49ea-b198-0b4d1d79a517\") " pod="openshift-network-operator/iptables-alerter-66v2k" Aug 04 17:52:26.800701 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800327 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-var-lib-openvswitch\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.800701 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800353 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b3f65ebe-8341-406d-bc45-a6e176fccd97-agent-certs\") pod \"konnectivity-agent-64c4r\" (UID: \"b3f65ebe-8341-406d-bc45-a6e176fccd97\") " pod="kube-system/konnectivity-agent-64c4r" Aug 04 17:52:26.800701 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800379 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-system-cni-dir\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.800701 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800415 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c600bdf9-3cef-457a-bb0b-704c5856c077-tmp-dir\") pod \"node-resolver-4dgxz\" (UID: \"c600bdf9-3cef-457a-bb0b-704c5856c077\") " pod="openshift-dns/node-resolver-4dgxz" Aug 04 17:52:26.800701 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800437 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/610b5dc3-b664-4744-b311-85bbd40575b4-socket-dir\") pod \"aws-ebs-csi-driver-node-57kll\" (UID: \"610b5dc3-b664-4744-b311-85bbd40575b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" Aug 04 17:52:26.800701 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800437 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c11d0415-39ac-4682-81d2-130d039fae98-env-overrides\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.800701 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800504 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-run-openvswitch\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.800701 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800530 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7pnh\" (UniqueName: \"kubernetes.io/projected/a1dbfaf6-9a2f-4146-807c-418181c922c1-kube-api-access-k7pnh\") pod \"multus-additional-cni-plugins-t2nnw\" (UID: \"a1dbfaf6-9a2f-4146-807c-418181c922c1\") " pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.800701 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800565 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fbeee23-8c17-49ea-b198-0b4d1d79a517-host-slash\") pod \"iptables-alerter-66v2k\" (UID: \"4fbeee23-8c17-49ea-b198-0b4d1d79a517\") " pod="openshift-network-operator/iptables-alerter-66v2k" Aug 04 17:52:26.800701 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800593 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plxpt\" (UniqueName: \"kubernetes.io/projected/8d005c33-5c1e-4ec8-990d-ee4d27743422-kube-api-access-plxpt\") pod \"node-ca-6ns66\" (UID: \"8d005c33-5c1e-4ec8-990d-ee4d27743422\") " pod="openshift-image-registry/node-ca-6ns66" Aug 04 17:52:26.800701 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800624 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-host-run-ovn-kubernetes\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.800701 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800657 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/096b3bbb-ead1-451f-9f3a-e5a4add161ad-kubelet-config\") pod \"global-pull-secret-syncer-xbhvg\" (UID: \"096b3bbb-ead1-451f-9f3a-e5a4add161ad\") " pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:26.800701 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800699 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1dbfaf6-9a2f-4146-807c-418181c922c1-system-cni-dir\") pod \"multus-additional-cni-plugins-t2nnw\" (UID: \"a1dbfaf6-9a2f-4146-807c-418181c922c1\") " pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.801147 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800727 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1dbfaf6-9a2f-4146-807c-418181c922c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t2nnw\" (UID: \"a1dbfaf6-9a2f-4146-807c-418181c922c1\") " pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.801147 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800755 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcxxl\" (UniqueName: \"kubernetes.io/projected/64a17319-0103-46f7-95fb-e5b9369434ad-kube-api-access-lcxxl\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.801147 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800780 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d005c33-5c1e-4ec8-990d-ee4d27743422-host\") pod \"node-ca-6ns66\" (UID: \"8d005c33-5c1e-4ec8-990d-ee4d27743422\") " pod="openshift-image-registry/node-ca-6ns66" Aug 04 17:52:26.801147 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800810 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c11d0415-39ac-4682-81d2-130d039fae98-ovnkube-script-lib\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.801147 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800846 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/610b5dc3-b664-4744-b311-85bbd40575b4-socket-dir\") pod \"aws-ebs-csi-driver-node-57kll\" (UID: \"610b5dc3-b664-4744-b311-85bbd40575b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" Aug 04 17:52:26.801147 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800914 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-hostroot\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.801147 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800936 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-var-lib-kubelet\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.801147 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800960 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/610b5dc3-b664-4744-b311-85bbd40575b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-57kll\" (UID: \"610b5dc3-b664-4744-b311-85bbd40575b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" Aug 04 17:52:26.801147 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.800991 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d005c33-5c1e-4ec8-990d-ee4d27743422-serviceca\") pod \"node-ca-6ns66\" (UID: \"8d005c33-5c1e-4ec8-990d-ee4d27743422\") " pod="openshift-image-registry/node-ca-6ns66" Aug 04 17:52:26.801147 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801007 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64a17319-0103-46f7-95fb-e5b9369434ad-cni-binary-copy\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.801147 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801024 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-sys\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.801147 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801040 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/610b5dc3-b664-4744-b311-85bbd40575b4-device-dir\") pod \"aws-ebs-csi-driver-node-57kll\" (UID: \"610b5dc3-b664-4744-b311-85bbd40575b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" Aug 04 17:52:26.801147 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801059 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-node-log\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.801147 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801087 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wg98\" (UniqueName: \"kubernetes.io/projected/c11d0415-39ac-4682-81d2-130d039fae98-kube-api-access-2wg98\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.801147 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801105 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/096b3bbb-ead1-451f-9f3a-e5a4add161ad-original-pull-secret\") pod \"global-pull-secret-syncer-xbhvg\" (UID: \"096b3bbb-ead1-451f-9f3a-e5a4add161ad\") " pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:26.801147 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801124 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1dbfaf6-9a2f-4146-807c-418181c922c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-t2nnw\" (UID: \"a1dbfaf6-9a2f-4146-807c-418181c922c1\") " pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.801147 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801140 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-host-var-lib-cni-bin\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.801691 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801158 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-etc-sysctl-conf\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.801691 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801182 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-etc-sysconfig\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.801691 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801211 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-etc-systemd\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.801691 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801227 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-run\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.801691 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801242 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3616edbe-d877-4980-b66a-e96223d78562-etc-tuned\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.801691 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801258 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-host-kubelet\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.801691 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801274 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-host-cni-netd\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.801691 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801289 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-host\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.801691 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801305 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g74lb\" (UniqueName: \"kubernetes.io/projected/c600bdf9-3cef-457a-bb0b-704c5856c077-kube-api-access-g74lb\") pod \"node-resolver-4dgxz\" (UID: \"c600bdf9-3cef-457a-bb0b-704c5856c077\") " pod="openshift-dns/node-resolver-4dgxz" Aug 04 17:52:26.801691 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801320 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/096b3bbb-ead1-451f-9f3a-e5a4add161ad-dbus\") pod \"global-pull-secret-syncer-xbhvg\" (UID: \"096b3bbb-ead1-451f-9f3a-e5a4add161ad\") " pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:26.801691 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801494 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c11d0415-39ac-4682-81d2-130d039fae98-ovnkube-script-lib\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.801691 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801599 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c600bdf9-3cef-457a-bb0b-704c5856c077-hosts-file\") pod \"node-resolver-4dgxz\" (UID: \"c600bdf9-3cef-457a-bb0b-704c5856c077\") " pod="openshift-dns/node-resolver-4dgxz" Aug 04 17:52:26.801691 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801620 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/610b5dc3-b664-4744-b311-85bbd40575b4-registration-dir\") pod \"aws-ebs-csi-driver-node-57kll\" (UID: \"610b5dc3-b664-4744-b311-85bbd40575b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" Aug 04 17:52:26.801691 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801638 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c11d0415-39ac-4682-81d2-130d039fae98-ovn-node-metrics-cert\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.801691 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801658 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aa144f2-e9b4-4162-be3f-6cd9405ac466-metrics-certs\") pod \"network-metrics-daemon-mpgkn\" (UID: \"8aa144f2-e9b4-4162-be3f-6cd9405ac466\") " pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:26.801691 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801675 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-host-var-lib-kubelet\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.802331 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801694 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-host-run-multus-certs\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.802331 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801739 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-etc-sysctl-d\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.802331 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801790 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzh56\" (UniqueName: \"kubernetes.io/projected/3616edbe-d877-4980-b66a-e96223d78562-kube-api-access-gzh56\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.802331 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801812 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-host-slash\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.802331 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801841 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-host-run-netns\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.802331 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801862 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-run-systemd\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.802331 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801905 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1dbfaf6-9a2f-4146-807c-418181c922c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t2nnw\" (UID: \"a1dbfaf6-9a2f-4146-807c-418181c922c1\") " pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.802331 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801929 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-etc-kubernetes\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.802331 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801961 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-systemd-units\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.802331 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801989 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-log-socket\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.802331 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802009 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-var-lib-openvswitch\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.802331 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802020 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c11d0415-39ac-4682-81d2-130d039fae98-ovnkube-config\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.802331 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802047 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b3f65ebe-8341-406d-bc45-a6e176fccd97-konnectivity-ca\") pod \"konnectivity-agent-64c4r\" (UID: \"b3f65ebe-8341-406d-bc45-a6e176fccd97\") " pod="kube-system/konnectivity-agent-64c4r" Aug 04 17:52:26.802331 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802092 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlhnn\" (UniqueName: \"kubernetes.io/projected/8aa144f2-e9b4-4162-be3f-6cd9405ac466-kube-api-access-hlhnn\") pod \"network-metrics-daemon-mpgkn\" (UID: \"8aa144f2-e9b4-4162-be3f-6cd9405ac466\") " pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:26.802331 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802129 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-cnibin\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.802331 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802156 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-host-run-k8s-cni-cncf-io\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.802331 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802185 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-multus-conf-dir\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.803038 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802219 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3616edbe-d877-4980-b66a-e96223d78562-tmp\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.803038 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802254 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/610b5dc3-b664-4744-b311-85bbd40575b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-57kll\" (UID: \"610b5dc3-b664-4744-b311-85bbd40575b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" Aug 04 17:52:26.803038 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802282 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/610b5dc3-b664-4744-b311-85bbd40575b4-sys-fs\") pod \"aws-ebs-csi-driver-node-57kll\" (UID: \"610b5dc3-b664-4744-b311-85bbd40575b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" Aug 04 17:52:26.803038 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802296 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d005c33-5c1e-4ec8-990d-ee4d27743422-host\") pod \"node-ca-6ns66\" (UID: \"8d005c33-5c1e-4ec8-990d-ee4d27743422\") " pod="openshift-image-registry/node-ca-6ns66" Aug 04 17:52:26.803038 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802300 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c600bdf9-3cef-457a-bb0b-704c5856c077-tmp-dir\") pod \"node-resolver-4dgxz\" (UID: \"c600bdf9-3cef-457a-bb0b-704c5856c077\") " pod="openshift-dns/node-resolver-4dgxz" Aug 04 17:52:26.803038 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802323 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-etc-openvswitch\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.803038 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802347 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fbeee23-8c17-49ea-b198-0b4d1d79a517-host-slash\") pod \"iptables-alerter-66v2k\" (UID: \"4fbeee23-8c17-49ea-b198-0b4d1d79a517\") " pod="openshift-network-operator/iptables-alerter-66v2k" Aug 04 17:52:26.803038 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802355 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-run-ovn\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.803038 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.801599 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-etc-modprobe-d\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.803038 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802458 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/610b5dc3-b664-4744-b311-85bbd40575b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-57kll\" (UID: \"610b5dc3-b664-4744-b311-85bbd40575b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" Aug 04 17:52:26.803038 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802485 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-host-run-ovn-kubernetes\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.803038 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802503 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-var-lib-kubelet\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.803038 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802528 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c600bdf9-3cef-457a-bb0b-704c5856c077-hosts-file\") pod \"node-resolver-4dgxz\" (UID: \"c600bdf9-3cef-457a-bb0b-704c5856c077\") " pod="openshift-dns/node-resolver-4dgxz" Aug 04 17:52:26.803038 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802536 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1dbfaf6-9a2f-4146-807c-418181c922c1-cnibin\") pod \"multus-additional-cni-plugins-t2nnw\" (UID: \"a1dbfaf6-9a2f-4146-807c-418181c922c1\") " pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.803038 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802590 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-host-run-netns\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.803038 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802603 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/610b5dc3-b664-4744-b311-85bbd40575b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-57kll\" (UID: \"610b5dc3-b664-4744-b311-85bbd40575b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" Aug 04 17:52:26.803038 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802642 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/64a17319-0103-46f7-95fb-e5b9369434ad-multus-daemon-config\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.803038 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802734 2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 04 17:52:26.803922 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802792 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-log-socket\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.803922 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802796 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-systemd-units\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.803922 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802822 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-sys\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.803922 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802887 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/610b5dc3-b664-4744-b311-85bbd40575b4-device-dir\") pod \"aws-ebs-csi-driver-node-57kll\" (UID: \"610b5dc3-b664-4744-b311-85bbd40575b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" Aug 04 17:52:26.803922 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802896 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-host-run-netns\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.803922 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802948 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-node-log\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.803922 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.803030 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-etc-sysctl-d\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.803922 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.803070 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8d005c33-5c1e-4ec8-990d-ee4d27743422-serviceca\") pod \"node-ca-6ns66\" (UID: \"8d005c33-5c1e-4ec8-990d-ee4d27743422\") " pod="openshift-image-registry/node-ca-6ns66" Aug 04 17:52:26.803922 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.803101 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/610b5dc3-b664-4744-b311-85bbd40575b4-sys-fs\") pod \"aws-ebs-csi-driver-node-57kll\" (UID: \"610b5dc3-b664-4744-b311-85bbd40575b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" Aug 04 17:52:26.803922 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.803139 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-etc-openvswitch\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.803922 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.803164 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c11d0415-39ac-4682-81d2-130d039fae98-ovnkube-config\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.803922 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.803235 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/610b5dc3-b664-4744-b311-85bbd40575b4-registration-dir\") pod \"aws-ebs-csi-driver-node-57kll\" (UID: \"610b5dc3-b664-4744-b311-85bbd40575b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" Aug 04 17:52:26.803922 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.803250 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-etc-sysconfig\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.803922 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.803278 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-host\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.803922 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.803291 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-host-kubelet\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.803922 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.803297 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-host-slash\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.803922 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.802764 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4fbeee23-8c17-49ea-b198-0b4d1d79a517-iptables-alerter-script\") pod \"iptables-alerter-66v2k\" (UID: \"4fbeee23-8c17-49ea-b198-0b4d1d79a517\") " pod="openshift-network-operator/iptables-alerter-66v2k" Aug 04 17:52:26.803922 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.803337 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-run-openvswitch\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.804623 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.803353 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-run\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.804623 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.803399 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-etc-sysctl-conf\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.804623 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.803406 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3616edbe-d877-4980-b66a-e96223d78562-etc-systemd\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.804623 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.803410 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-run-systemd\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.804623 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.803449 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-run-ovn\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.804623 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.803452 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c11d0415-39ac-4682-81d2-130d039fae98-host-cni-netd\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.806811 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.806789 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3616edbe-d877-4980-b66a-e96223d78562-etc-tuned\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.806944 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.806834 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3616edbe-d877-4980-b66a-e96223d78562-tmp\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.807521 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.807496 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c11d0415-39ac-4682-81d2-130d039fae98-ovn-node-metrics-cert\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.810277 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.810250 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qttzb\" (UniqueName: \"kubernetes.io/projected/610b5dc3-b664-4744-b311-85bbd40575b4-kube-api-access-qttzb\") pod \"aws-ebs-csi-driver-node-57kll\" (UID: \"610b5dc3-b664-4744-b311-85bbd40575b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" Aug 04 17:52:26.810458 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.810424 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s74b\" (UniqueName: \"kubernetes.io/projected/4fbeee23-8c17-49ea-b198-0b4d1d79a517-kube-api-access-8s74b\") pod \"iptables-alerter-66v2k\" (UID: \"4fbeee23-8c17-49ea-b198-0b4d1d79a517\") " pod="openshift-network-operator/iptables-alerter-66v2k" Aug 04 17:52:26.811733 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.811711 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g74lb\" (UniqueName: \"kubernetes.io/projected/c600bdf9-3cef-457a-bb0b-704c5856c077-kube-api-access-g74lb\") pod \"node-resolver-4dgxz\" (UID: \"c600bdf9-3cef-457a-bb0b-704c5856c077\") " pod="openshift-dns/node-resolver-4dgxz" Aug 04 17:52:26.814498 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.814471 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plxpt\" (UniqueName: \"kubernetes.io/projected/8d005c33-5c1e-4ec8-990d-ee4d27743422-kube-api-access-plxpt\") pod \"node-ca-6ns66\" (UID: \"8d005c33-5c1e-4ec8-990d-ee4d27743422\") " pod="openshift-image-registry/node-ca-6ns66" Aug 04 17:52:26.815343 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.815326 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wg98\" (UniqueName: \"kubernetes.io/projected/c11d0415-39ac-4682-81d2-130d039fae98-kube-api-access-2wg98\") pod \"ovnkube-node-2lkw4\" (UID: \"c11d0415-39ac-4682-81d2-130d039fae98\") " pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.815589 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.815568 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzh56\" (UniqueName: \"kubernetes.io/projected/3616edbe-d877-4980-b66a-e96223d78562-kube-api-access-gzh56\") pod \"tuned-dt2dw\" (UID: \"3616edbe-d877-4980-b66a-e96223d78562\") " pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:26.825417 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.825358 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" event={"ID":"9ea37b25d2fb147b8b95f79511393014","Type":"ContainerStarted","Data":"57bf9de02cb3c1a278e51d1366415f2e9ee46b3a71c00da82743d2a99b7c0be9"} Aug 04 17:52:26.826838 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.826809 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-173.ec2.internal" event={"ID":"37a95ca805868321653a87ed06a2c559","Type":"ContainerStarted","Data":"82e1722474992a308aa100e4e6aa333c2e52d4861093c74522f9873d885d7539"} Aug 04 17:52:26.903970 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.903935 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64a17319-0103-46f7-95fb-e5b9369434ad-cni-binary-copy\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.903970 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.903982 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/096b3bbb-ead1-451f-9f3a-e5a4add161ad-original-pull-secret\") pod \"global-pull-secret-syncer-xbhvg\" (UID: \"096b3bbb-ead1-451f-9f3a-e5a4add161ad\") " pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:26.904197 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904011 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1dbfaf6-9a2f-4146-807c-418181c922c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-t2nnw\" (UID: \"a1dbfaf6-9a2f-4146-807c-418181c922c1\") " pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.904197 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904042 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-host-var-lib-cni-bin\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.904197 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904078 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/096b3bbb-ead1-451f-9f3a-e5a4add161ad-dbus\") pod \"global-pull-secret-syncer-xbhvg\" (UID: \"096b3bbb-ead1-451f-9f3a-e5a4add161ad\") " pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:26.904197 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904112 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aa144f2-e9b4-4162-be3f-6cd9405ac466-metrics-certs\") pod \"network-metrics-daemon-mpgkn\" (UID: \"8aa144f2-e9b4-4162-be3f-6cd9405ac466\") " pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:26.904197 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:26.904114 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:26.904197 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904131 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-host-var-lib-kubelet\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.904197 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904156 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-host-run-multus-certs\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.904554 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:26.904204 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096b3bbb-ead1-451f-9f3a-e5a4add161ad-original-pull-secret podName:096b3bbb-ead1-451f-9f3a-e5a4add161ad nodeName:}" failed. No retries permitted until 2026-08-04 17:52:27.404177561 +0000 UTC m=+3.094868609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/096b3bbb-ead1-451f-9f3a-e5a4add161ad-original-pull-secret") pod "global-pull-secret-syncer-xbhvg" (UID: "096b3bbb-ead1-451f-9f3a-e5a4add161ad") : object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:26.904554 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904232 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1dbfaf6-9a2f-4146-807c-418181c922c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t2nnw\" (UID: \"a1dbfaf6-9a2f-4146-807c-418181c922c1\") " pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.904554 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904265 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-etc-kubernetes\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.904554 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904300 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b3f65ebe-8341-406d-bc45-a6e176fccd97-konnectivity-ca\") pod \"konnectivity-agent-64c4r\" (UID: \"b3f65ebe-8341-406d-bc45-a6e176fccd97\") " pod="kube-system/konnectivity-agent-64c4r" Aug 04 17:52:26.904554 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904325 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlhnn\" (UniqueName: \"kubernetes.io/projected/8aa144f2-e9b4-4162-be3f-6cd9405ac466-kube-api-access-hlhnn\") pod \"network-metrics-daemon-mpgkn\" (UID: \"8aa144f2-e9b4-4162-be3f-6cd9405ac466\") " pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:26.904554 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904352 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-cnibin\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.904554 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904377 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-host-run-k8s-cni-cncf-io\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.904554 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904402 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-multus-conf-dir\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.904554 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904432 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1dbfaf6-9a2f-4146-807c-418181c922c1-cnibin\") pod \"multus-additional-cni-plugins-t2nnw\" (UID: \"a1dbfaf6-9a2f-4146-807c-418181c922c1\") " pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.904554 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904445 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-host-var-lib-kubelet\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.904554 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904516 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-host-run-netns\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.904554 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904545 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/096b3bbb-ead1-451f-9f3a-e5a4add161ad-dbus\") pod \"global-pull-secret-syncer-xbhvg\" (UID: \"096b3bbb-ead1-451f-9f3a-e5a4add161ad\") " pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:26.904997 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904602 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-host-run-multus-certs\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.904997 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:26.904613 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:26.904997 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904621 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64a17319-0103-46f7-95fb-e5b9369434ad-cni-binary-copy\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.904997 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904626 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1dbfaf6-9a2f-4146-807c-418181c922c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-t2nnw\" (UID: \"a1dbfaf6-9a2f-4146-807c-418181c922c1\") " pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.904997 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904665 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-host-var-lib-cni-bin\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.904997 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904457 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-host-run-netns\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.904997 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:26.904676 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa144f2-e9b4-4162-be3f-6cd9405ac466-metrics-certs podName:8aa144f2-e9b4-4162-be3f-6cd9405ac466 nodeName:}" failed. No retries permitted until 2026-08-04 17:52:27.404659627 +0000 UTC m=+3.095350662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8aa144f2-e9b4-4162-be3f-6cd9405ac466-metrics-certs") pod "network-metrics-daemon-mpgkn" (UID: "8aa144f2-e9b4-4162-be3f-6cd9405ac466") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:26.904997 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904691 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-host-run-k8s-cni-cncf-io\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.904997 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904700 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-etc-kubernetes\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.904997 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904729 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-multus-conf-dir\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.904997 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904715 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/64a17319-0103-46f7-95fb-e5b9369434ad-multus-daemon-config\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.904997 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904773 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a1dbfaf6-9a2f-4146-807c-418181c922c1-os-release\") pod \"multus-additional-cni-plugins-t2nnw\" (UID: \"a1dbfaf6-9a2f-4146-807c-418181c922c1\") " pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.904997 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904787 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-cnibin\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.904997 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904798 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-multus-socket-dir-parent\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.904997 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904812 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1dbfaf6-9a2f-4146-807c-418181c922c1-cnibin\") pod \"multus-additional-cni-plugins-t2nnw\" (UID: \"a1dbfaf6-9a2f-4146-807c-418181c922c1\") " pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.904997 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904829 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1dbfaf6-9a2f-4146-807c-418181c922c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t2nnw\" (UID: \"a1dbfaf6-9a2f-4146-807c-418181c922c1\") " pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.904997 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904856 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-multus-cni-dir\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.905570 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904873 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a1dbfaf6-9a2f-4146-807c-418181c922c1-os-release\") pod \"multus-additional-cni-plugins-t2nnw\" (UID: \"a1dbfaf6-9a2f-4146-807c-418181c922c1\") " pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.905570 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904881 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-os-release\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.905570 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904884 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1dbfaf6-9a2f-4146-807c-418181c922c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t2nnw\" (UID: \"a1dbfaf6-9a2f-4146-807c-418181c922c1\") " pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.905570 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904909 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-479kg\" (UniqueName: \"kubernetes.io/projected/4bb95b54-5442-43b6-befa-8ecf7b8f4781-kube-api-access-479kg\") pod \"network-check-target-9z2cq\" (UID: \"4bb95b54-5442-43b6-befa-8ecf7b8f4781\") " pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:26.905570 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.904981 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-multus-socket-dir-parent\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.905570 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.905105 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-multus-cni-dir\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.905570 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.905146 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-host-var-lib-cni-multus\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.905570 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.905184 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b3f65ebe-8341-406d-bc45-a6e176fccd97-agent-certs\") pod \"konnectivity-agent-64c4r\" (UID: \"b3f65ebe-8341-406d-bc45-a6e176fccd97\") " pod="kube-system/konnectivity-agent-64c4r" Aug 04 17:52:26.905570 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.905236 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-system-cni-dir\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.905570 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.905251 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b3f65ebe-8341-406d-bc45-a6e176fccd97-konnectivity-ca\") pod \"konnectivity-agent-64c4r\" (UID: \"b3f65ebe-8341-406d-bc45-a6e176fccd97\") " pod="kube-system/konnectivity-agent-64c4r" Aug 04 17:52:26.905570 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.905267 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7pnh\" (UniqueName: \"kubernetes.io/projected/a1dbfaf6-9a2f-4146-807c-418181c922c1-kube-api-access-k7pnh\") pod \"multus-additional-cni-plugins-t2nnw\" (UID: \"a1dbfaf6-9a2f-4146-807c-418181c922c1\") " pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.905570 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.905300 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/096b3bbb-ead1-451f-9f3a-e5a4add161ad-kubelet-config\") pod \"global-pull-secret-syncer-xbhvg\" (UID: \"096b3bbb-ead1-451f-9f3a-e5a4add161ad\") " pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:26.905570 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.905324 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1dbfaf6-9a2f-4146-807c-418181c922c1-system-cni-dir\") pod \"multus-additional-cni-plugins-t2nnw\" (UID: \"a1dbfaf6-9a2f-4146-807c-418181c922c1\") " pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.905570 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.905352 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1dbfaf6-9a2f-4146-807c-418181c922c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t2nnw\" (UID: \"a1dbfaf6-9a2f-4146-807c-418181c922c1\") " pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.905570 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.905382 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcxxl\" (UniqueName: \"kubernetes.io/projected/64a17319-0103-46f7-95fb-e5b9369434ad-kube-api-access-lcxxl\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.905570 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.905410 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-hostroot\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.905570 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.905413 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-host-var-lib-cni-multus\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.906302 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.905464 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1dbfaf6-9a2f-4146-807c-418181c922c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t2nnw\" (UID: \"a1dbfaf6-9a2f-4146-807c-418181c922c1\") " pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.906302 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.905485 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/096b3bbb-ead1-451f-9f3a-e5a4add161ad-kubelet-config\") pod \"global-pull-secret-syncer-xbhvg\" (UID: \"096b3bbb-ead1-451f-9f3a-e5a4add161ad\") " pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:26.906302 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.905533 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1dbfaf6-9a2f-4146-807c-418181c922c1-system-cni-dir\") pod \"multus-additional-cni-plugins-t2nnw\" (UID: \"a1dbfaf6-9a2f-4146-807c-418181c922c1\") " pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.906302 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.905538 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-hostroot\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.906302 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.905382 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/64a17319-0103-46f7-95fb-e5b9369434ad-multus-daemon-config\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.906302 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.905730 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-os-release\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.906302 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.905889 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64a17319-0103-46f7-95fb-e5b9369434ad-system-cni-dir\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.906302 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.906004 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1dbfaf6-9a2f-4146-807c-418181c922c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t2nnw\" (UID: \"a1dbfaf6-9a2f-4146-807c-418181c922c1\") " pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.908507 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.908480 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b3f65ebe-8341-406d-bc45-a6e176fccd97-agent-certs\") pod \"konnectivity-agent-64c4r\" (UID: \"b3f65ebe-8341-406d-bc45-a6e176fccd97\") " pod="kube-system/konnectivity-agent-64c4r" Aug 04 17:52:26.915166 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:26.915091 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 17:52:26.915166 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:26.915122 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 17:52:26.915166 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:26.915144 2572 projected.go:196] Error preparing data for projected volume kube-api-access-479kg for pod openshift-network-diagnostics/network-check-target-9z2cq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:26.915398 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:26.915272 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bb95b54-5442-43b6-befa-8ecf7b8f4781-kube-api-access-479kg podName:4bb95b54-5442-43b6-befa-8ecf7b8f4781 nodeName:}" failed. No retries permitted until 2026-08-04 17:52:27.415252827 +0000 UTC m=+3.105943862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-479kg" (UniqueName: "kubernetes.io/projected/4bb95b54-5442-43b6-befa-8ecf7b8f4781-kube-api-access-479kg") pod "network-check-target-9z2cq" (UID: "4bb95b54-5442-43b6-befa-8ecf7b8f4781") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:26.916780 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.916716 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlhnn\" (UniqueName: \"kubernetes.io/projected/8aa144f2-e9b4-4162-be3f-6cd9405ac466-kube-api-access-hlhnn\") pod \"network-metrics-daemon-mpgkn\" (UID: \"8aa144f2-e9b4-4162-be3f-6cd9405ac466\") " pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:26.918054 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.918030 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcxxl\" (UniqueName: \"kubernetes.io/projected/64a17319-0103-46f7-95fb-e5b9369434ad-kube-api-access-lcxxl\") pod \"multus-qqgm5\" (UID: \"64a17319-0103-46f7-95fb-e5b9369434ad\") " pod="openshift-multus/multus-qqgm5" Aug 04 17:52:26.920676 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.920650 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7pnh\" (UniqueName: \"kubernetes.io/projected/a1dbfaf6-9a2f-4146-807c-418181c922c1-kube-api-access-k7pnh\") pod \"multus-additional-cni-plugins-t2nnw\" (UID: \"a1dbfaf6-9a2f-4146-807c-418181c922c1\") " pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:26.955687 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.955634 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 17:52:26.989182 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.989141 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:26.997543 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:26.997522 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" Aug 04 17:52:27.016355 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:27.016317 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-66v2k" Aug 04 17:52:27.023272 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:27.023252 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4dgxz" Aug 04 17:52:27.033037 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:27.033014 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" Aug 04 17:52:27.038830 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:27.038808 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6ns66" Aug 04 17:52:27.046669 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:27.046650 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t2nnw" Aug 04 17:52:27.055566 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:27.055540 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qqgm5" Aug 04 17:52:27.062379 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:27.062356 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-64c4r" Aug 04 17:52:27.410135 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:27.410100 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/096b3bbb-ead1-451f-9f3a-e5a4add161ad-original-pull-secret\") pod \"global-pull-secret-syncer-xbhvg\" (UID: \"096b3bbb-ead1-451f-9f3a-e5a4add161ad\") " pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:27.410289 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:27.410152 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aa144f2-e9b4-4162-be3f-6cd9405ac466-metrics-certs\") pod \"network-metrics-daemon-mpgkn\" (UID: \"8aa144f2-e9b4-4162-be3f-6cd9405ac466\") " pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:27.410344 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:27.410277 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:27.410406 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:27.410351 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096b3bbb-ead1-451f-9f3a-e5a4add161ad-original-pull-secret podName:096b3bbb-ead1-451f-9f3a-e5a4add161ad nodeName:}" failed. No retries permitted until 2026-08-04 17:52:28.410335459 +0000 UTC m=+4.101026494 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/096b3bbb-ead1-451f-9f3a-e5a4add161ad-original-pull-secret") pod "global-pull-secret-syncer-xbhvg" (UID: "096b3bbb-ead1-451f-9f3a-e5a4add161ad") : object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:27.410406 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:27.410283 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:27.410406 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:27.410399 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa144f2-e9b4-4162-be3f-6cd9405ac466-metrics-certs podName:8aa144f2-e9b4-4162-be3f-6cd9405ac466 nodeName:}" failed. No retries permitted until 2026-08-04 17:52:28.410391563 +0000 UTC m=+4.101082594 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8aa144f2-e9b4-4162-be3f-6cd9405ac466-metrics-certs") pod "network-metrics-daemon-mpgkn" (UID: "8aa144f2-e9b4-4162-be3f-6cd9405ac466") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:27.511635 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:27.511596 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-479kg\" (UniqueName: \"kubernetes.io/projected/4bb95b54-5442-43b6-befa-8ecf7b8f4781-kube-api-access-479kg\") pod \"network-check-target-9z2cq\" (UID: \"4bb95b54-5442-43b6-befa-8ecf7b8f4781\") " pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:27.511830 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:27.511809 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 17:52:27.511830 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:27.511827 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 17:52:27.511923 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:27.511837 2572 projected.go:196] Error preparing data for projected volume kube-api-access-479kg for pod openshift-network-diagnostics/network-check-target-9z2cq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:27.511923 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:27.511901 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bb95b54-5442-43b6-befa-8ecf7b8f4781-kube-api-access-479kg podName:4bb95b54-5442-43b6-befa-8ecf7b8f4781 nodeName:}" failed. No retries permitted until 2026-08-04 17:52:28.511884615 +0000 UTC m=+4.202575646 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-479kg" (UniqueName: "kubernetes.io/projected/4bb95b54-5442-43b6-befa-8ecf7b8f4781-kube-api-access-479kg") pod "network-check-target-9z2cq" (UID: "4bb95b54-5442-43b6-befa-8ecf7b8f4781") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:27.735151 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:27.735092 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 17:47:27 +0000 UTC" deadline="2028-03-17 08:03:18.239520767 +0000 UTC" Aug 04 17:52:27.735151 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:27.735142 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14174h10m50.504384293s" Aug 04 17:52:27.816100 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:27.816070 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:27.816263 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:27.816199 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:52:27.941205 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:27.941176 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11d0415_39ac_4682_81d2_130d039fae98.slice/crio-1f834a49e7e518f89bb8b75ac4b73c5b7e28c83478281b9da5161dd6440d7697 WatchSource:0}: Error finding container 1f834a49e7e518f89bb8b75ac4b73c5b7e28c83478281b9da5161dd6440d7697: Status 404 returned error can't find the container with id 1f834a49e7e518f89bb8b75ac4b73c5b7e28c83478281b9da5161dd6440d7697 Aug 04 17:52:27.947460 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:27.946951 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc600bdf9_3cef_457a_bb0b_704c5856c077.slice/crio-54dbc7f965fb771044d7a07e819089c8c62ed435898ff8f2808c63de3636ec25 WatchSource:0}: Error finding container 54dbc7f965fb771044d7a07e819089c8c62ed435898ff8f2808c63de3636ec25: Status 404 returned error can't find the container with id 54dbc7f965fb771044d7a07e819089c8c62ed435898ff8f2808c63de3636ec25 Aug 04 17:52:27.953685 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:52:27.952826 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610b5dc3_b664_4744_b311_85bbd40575b4.slice/crio-49a4057cb6a2d5cc494c17b903c378e0998260a2d3214b31db58d226f5ffbc0a WatchSource:0}: Error finding container 49a4057cb6a2d5cc494c17b903c378e0998260a2d3214b31db58d226f5ffbc0a: Status 404 returned error can't find the container with id 49a4057cb6a2d5cc494c17b903c378e0998260a2d3214b31db58d226f5ffbc0a Aug 04 17:52:28.418347 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:28.418087 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aa144f2-e9b4-4162-be3f-6cd9405ac466-metrics-certs\") pod \"network-metrics-daemon-mpgkn\" (UID: \"8aa144f2-e9b4-4162-be3f-6cd9405ac466\") " pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:28.418347 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:28.418233 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/096b3bbb-ead1-451f-9f3a-e5a4add161ad-original-pull-secret\") pod \"global-pull-secret-syncer-xbhvg\" (UID: \"096b3bbb-ead1-451f-9f3a-e5a4add161ad\") " pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:28.418646 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:28.418425 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:28.418646 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:28.418495 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096b3bbb-ead1-451f-9f3a-e5a4add161ad-original-pull-secret podName:096b3bbb-ead1-451f-9f3a-e5a4add161ad nodeName:}" failed. No retries permitted until 2026-08-04 17:52:30.418474893 +0000 UTC m=+6.109165930 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/096b3bbb-ead1-451f-9f3a-e5a4add161ad-original-pull-secret") pod "global-pull-secret-syncer-xbhvg" (UID: "096b3bbb-ead1-451f-9f3a-e5a4add161ad") : object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:28.419168 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:28.418969 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:28.419168 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:28.419111 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa144f2-e9b4-4162-be3f-6cd9405ac466-metrics-certs podName:8aa144f2-e9b4-4162-be3f-6cd9405ac466 nodeName:}" failed. No retries permitted until 2026-08-04 17:52:30.419081964 +0000 UTC m=+6.109773022 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8aa144f2-e9b4-4162-be3f-6cd9405ac466-metrics-certs") pod "network-metrics-daemon-mpgkn" (UID: "8aa144f2-e9b4-4162-be3f-6cd9405ac466") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:28.519015 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:28.518930 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-479kg\" (UniqueName: \"kubernetes.io/projected/4bb95b54-5442-43b6-befa-8ecf7b8f4781-kube-api-access-479kg\") pod \"network-check-target-9z2cq\" (UID: \"4bb95b54-5442-43b6-befa-8ecf7b8f4781\") " pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:28.519180 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:28.519160 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 17:52:28.519255 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:28.519188 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 17:52:28.519255 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:28.519203 2572 projected.go:196] Error preparing data for projected volume kube-api-access-479kg for pod openshift-network-diagnostics/network-check-target-9z2cq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:28.519366 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:28.519263 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bb95b54-5442-43b6-befa-8ecf7b8f4781-kube-api-access-479kg podName:4bb95b54-5442-43b6-befa-8ecf7b8f4781 nodeName:}" failed. No retries permitted until 2026-08-04 17:52:30.519244512 +0000 UTC m=+6.209935547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-479kg" (UniqueName: "kubernetes.io/projected/4bb95b54-5442-43b6-befa-8ecf7b8f4781-kube-api-access-479kg") pod "network-check-target-9z2cq" (UID: "4bb95b54-5442-43b6-befa-8ecf7b8f4781") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:28.735867 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:28.735765 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 17:47:27 +0000 UTC" deadline="2028-01-10 10:30:55.597953683 +0000 UTC" Aug 04 17:52:28.735867 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:28.735830 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12568h38m26.862128125s" Aug 04 17:52:28.819410 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:28.818628 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:28.819410 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:28.818791 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:52:28.819410 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:28.819209 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:28.819410 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:28.819303 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:52:28.850330 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:28.850284 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4dgxz" event={"ID":"c600bdf9-3cef-457a-bb0b-704c5856c077","Type":"ContainerStarted","Data":"54dbc7f965fb771044d7a07e819089c8c62ed435898ff8f2808c63de3636ec25"} Aug 04 17:52:28.855579 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:28.855542 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" event={"ID":"c11d0415-39ac-4682-81d2-130d039fae98","Type":"ContainerStarted","Data":"1f834a49e7e518f89bb8b75ac4b73c5b7e28c83478281b9da5161dd6440d7697"} Aug 04 17:52:28.866862 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:28.866515 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-173.ec2.internal" event={"ID":"37a95ca805868321653a87ed06a2c559","Type":"ContainerStarted","Data":"46000ea121e68f048ce7b219dfe5ac3250ba4c7063cbc8b779b92ad9d3eaf7e2"} Aug 04 17:52:28.873240 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:28.873209 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-64c4r" event={"ID":"b3f65ebe-8341-406d-bc45-a6e176fccd97","Type":"ContainerStarted","Data":"83f89e766d40768c35f8d1061779f54789ce4a41febb33d3bd2376d63b3565bf"} Aug 04 17:52:28.887470 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:28.887440 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qqgm5" event={"ID":"64a17319-0103-46f7-95fb-e5b9369434ad","Type":"ContainerStarted","Data":"ae0d18a54260751d7993e67b233d6f2a90a1bcaa960069bb76c98cc9759a30e7"} Aug 04 17:52:28.897392 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:28.897362 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" event={"ID":"610b5dc3-b664-4744-b311-85bbd40575b4","Type":"ContainerStarted","Data":"49a4057cb6a2d5cc494c17b903c378e0998260a2d3214b31db58d226f5ffbc0a"} Aug 04 17:52:28.913588 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:28.913561 2572 generic.go:356] "Generic (PLEG): container finished" podID="9ea37b25d2fb147b8b95f79511393014" containerID="8a00b13cea7b52796ecccde4deab62e64cc5f937687ccc299427e502228e2665" exitCode=0 Aug 04 17:52:28.913710 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:28.913640 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" event={"ID":"9ea37b25d2fb147b8b95f79511393014","Type":"ContainerDied","Data":"8a00b13cea7b52796ecccde4deab62e64cc5f937687ccc299427e502228e2665"} Aug 04 17:52:28.917481 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:28.917452 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2nnw" event={"ID":"a1dbfaf6-9a2f-4146-807c-418181c922c1","Type":"ContainerStarted","Data":"ad261874f67a56585163dc11bef2db88ab40394f320880a0e94184f779324148"} Aug 04 17:52:28.922630 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:28.922600 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-66v2k" event={"ID":"4fbeee23-8c17-49ea-b198-0b4d1d79a517","Type":"ContainerStarted","Data":"c3af025c8007e442582a958d88790aa35cc4d39708c4816cc6f8871819085c78"} Aug 04 17:52:28.926066 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:28.925986 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6ns66" event={"ID":"8d005c33-5c1e-4ec8-990d-ee4d27743422","Type":"ContainerStarted","Data":"7e43882560fa7c79130b58c898c1773802ed911b7b8d56b011bc358fc6735e21"} Aug 04 17:52:28.932432 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:28.932366 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" event={"ID":"3616edbe-d877-4980-b66a-e96223d78562","Type":"ContainerStarted","Data":"2e6d86a3399643843312cfae32cf0f5d2c85df4ffbd62865ffdfa96e2d83138b"} Aug 04 17:52:28.943852 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:28.942798 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-173.ec2.internal" podStartSLOduration=2.942711277 podStartE2EDuration="2.942711277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:52:28.889496436 +0000 UTC m=+4.580187495" watchObservedRunningTime="2026-08-04 17:52:28.942711277 +0000 UTC m=+4.633402324" Aug 04 17:52:29.815970 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:29.815912 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:29.816379 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:29.816064 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:52:29.953461 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:29.953403 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" event={"ID":"9ea37b25d2fb147b8b95f79511393014","Type":"ContainerStarted","Data":"5f3ac71d929a3b7b886d6d0d908fd953dd7204c9599f08cc3b7c11fae4bfab6a"} Aug 04 17:52:29.973465 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:29.973406 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" podStartSLOduration=3.973383851 podStartE2EDuration="3.973383851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:52:29.973050961 +0000 UTC m=+5.663742017" watchObservedRunningTime="2026-08-04 17:52:29.973383851 +0000 UTC m=+5.664074907" Aug 04 17:52:30.434649 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:30.434604 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/096b3bbb-ead1-451f-9f3a-e5a4add161ad-original-pull-secret\") pod \"global-pull-secret-syncer-xbhvg\" (UID: \"096b3bbb-ead1-451f-9f3a-e5a4add161ad\") " pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:30.434868 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:30.434675 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aa144f2-e9b4-4162-be3f-6cd9405ac466-metrics-certs\") pod \"network-metrics-daemon-mpgkn\" (UID: \"8aa144f2-e9b4-4162-be3f-6cd9405ac466\") " pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:30.434941 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:30.434898 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:30.434995 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:30.434974 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa144f2-e9b4-4162-be3f-6cd9405ac466-metrics-certs podName:8aa144f2-e9b4-4162-be3f-6cd9405ac466 nodeName:}" failed. No retries permitted until 2026-08-04 17:52:34.434946815 +0000 UTC m=+10.125637872 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8aa144f2-e9b4-4162-be3f-6cd9405ac466-metrics-certs") pod "network-metrics-daemon-mpgkn" (UID: "8aa144f2-e9b4-4162-be3f-6cd9405ac466") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:30.435419 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:30.435399 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:30.435503 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:30.435457 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096b3bbb-ead1-451f-9f3a-e5a4add161ad-original-pull-secret podName:096b3bbb-ead1-451f-9f3a-e5a4add161ad nodeName:}" failed. No retries permitted until 2026-08-04 17:52:34.435441992 +0000 UTC m=+10.126133063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/096b3bbb-ead1-451f-9f3a-e5a4add161ad-original-pull-secret") pod "global-pull-secret-syncer-xbhvg" (UID: "096b3bbb-ead1-451f-9f3a-e5a4add161ad") : object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:30.536154 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:30.536114 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-479kg\" (UniqueName: \"kubernetes.io/projected/4bb95b54-5442-43b6-befa-8ecf7b8f4781-kube-api-access-479kg\") pod \"network-check-target-9z2cq\" (UID: \"4bb95b54-5442-43b6-befa-8ecf7b8f4781\") " pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:30.536381 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:30.536358 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 17:52:30.536480 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:30.536401 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 17:52:30.536480 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:30.536417 2572 projected.go:196] Error preparing data for projected volume kube-api-access-479kg for pod openshift-network-diagnostics/network-check-target-9z2cq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:30.536604 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:30.536485 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bb95b54-5442-43b6-befa-8ecf7b8f4781-kube-api-access-479kg podName:4bb95b54-5442-43b6-befa-8ecf7b8f4781 nodeName:}" failed. No retries permitted until 2026-08-04 17:52:34.536466453 +0000 UTC m=+10.227157506 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-479kg" (UniqueName: "kubernetes.io/projected/4bb95b54-5442-43b6-befa-8ecf7b8f4781-kube-api-access-479kg") pod "network-check-target-9z2cq" (UID: "4bb95b54-5442-43b6-befa-8ecf7b8f4781") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:30.816917 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:30.816832 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:30.817430 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:30.816989 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:52:30.817430 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:30.817425 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:30.817548 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:30.817523 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:52:31.816450 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:31.816413 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:31.816634 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:31.816557 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:52:32.816559 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:32.816520 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:32.816999 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:32.816642 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:52:32.816999 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:32.816792 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:32.816999 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:32.816903 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:52:33.816492 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:33.816455 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:33.816684 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:33.816600 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:52:34.477089 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:34.477037 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/096b3bbb-ead1-451f-9f3a-e5a4add161ad-original-pull-secret\") pod \"global-pull-secret-syncer-xbhvg\" (UID: \"096b3bbb-ead1-451f-9f3a-e5a4add161ad\") " pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:34.477289 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:34.477094 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aa144f2-e9b4-4162-be3f-6cd9405ac466-metrics-certs\") pod \"network-metrics-daemon-mpgkn\" (UID: \"8aa144f2-e9b4-4162-be3f-6cd9405ac466\") " pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:34.477289 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:34.477275 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:34.477416 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:34.477370 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:34.477466 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:34.477439 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096b3bbb-ead1-451f-9f3a-e5a4add161ad-original-pull-secret podName:096b3bbb-ead1-451f-9f3a-e5a4add161ad nodeName:}" failed. No retries permitted until 2026-08-04 17:52:42.477416737 +0000 UTC m=+18.168107768 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/096b3bbb-ead1-451f-9f3a-e5a4add161ad-original-pull-secret") pod "global-pull-secret-syncer-xbhvg" (UID: "096b3bbb-ead1-451f-9f3a-e5a4add161ad") : object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:34.477664 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:34.477612 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa144f2-e9b4-4162-be3f-6cd9405ac466-metrics-certs podName:8aa144f2-e9b4-4162-be3f-6cd9405ac466 nodeName:}" failed. No retries permitted until 2026-08-04 17:52:42.477592276 +0000 UTC m=+18.168283310 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8aa144f2-e9b4-4162-be3f-6cd9405ac466-metrics-certs") pod "network-metrics-daemon-mpgkn" (UID: "8aa144f2-e9b4-4162-be3f-6cd9405ac466") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:34.578190 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:34.578152 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-479kg\" (UniqueName: \"kubernetes.io/projected/4bb95b54-5442-43b6-befa-8ecf7b8f4781-kube-api-access-479kg\") pod \"network-check-target-9z2cq\" (UID: \"4bb95b54-5442-43b6-befa-8ecf7b8f4781\") " pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:34.578378 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:34.578336 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 17:52:34.578378 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:34.578361 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 17:52:34.578378 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:34.578378 2572 projected.go:196] Error preparing data for projected volume kube-api-access-479kg for pod openshift-network-diagnostics/network-check-target-9z2cq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:34.578560 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:34.578445 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bb95b54-5442-43b6-befa-8ecf7b8f4781-kube-api-access-479kg podName:4bb95b54-5442-43b6-befa-8ecf7b8f4781 nodeName:}" failed. No retries permitted until 2026-08-04 17:52:42.578424215 +0000 UTC m=+18.269115267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-479kg" (UniqueName: "kubernetes.io/projected/4bb95b54-5442-43b6-befa-8ecf7b8f4781-kube-api-access-479kg") pod "network-check-target-9z2cq" (UID: "4bb95b54-5442-43b6-befa-8ecf7b8f4781") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:34.817448 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:34.817365 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:34.817982 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:34.817464 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:52:34.817982 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:34.817560 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:34.817982 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:34.817654 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:52:35.816789 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:35.816752 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:35.816960 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:35.816881 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:52:36.816809 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:36.816776 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:36.817279 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:36.816899 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:52:36.817417 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:36.817385 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:36.817525 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:36.817490 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:52:37.816755 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:37.816717 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:37.816941 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:37.816856 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:52:38.818627 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:38.818561 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:38.819061 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:38.818702 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:52:38.819061 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:38.818710 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:38.819061 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:38.818789 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:52:39.816755 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:39.816681 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:39.816901 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:39.816802 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:52:40.816120 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:40.816079 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:40.816523 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:40.816213 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:52:40.816523 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:40.816278 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:40.816523 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:40.816392 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:52:41.816550 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:41.816521 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:41.817130 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:41.816632 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:52:42.541820 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:42.541789 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/096b3bbb-ead1-451f-9f3a-e5a4add161ad-original-pull-secret\") pod \"global-pull-secret-syncer-xbhvg\" (UID: \"096b3bbb-ead1-451f-9f3a-e5a4add161ad\") " pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:42.542081 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:42.541839 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aa144f2-e9b4-4162-be3f-6cd9405ac466-metrics-certs\") pod \"network-metrics-daemon-mpgkn\" (UID: \"8aa144f2-e9b4-4162-be3f-6cd9405ac466\") " pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:42.542081 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:42.541969 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:42.542081 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:42.541966 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:42.542081 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:42.542027 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa144f2-e9b4-4162-be3f-6cd9405ac466-metrics-certs podName:8aa144f2-e9b4-4162-be3f-6cd9405ac466 nodeName:}" failed. No retries permitted until 2026-08-04 17:52:58.542008049 +0000 UTC m=+34.232699083 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8aa144f2-e9b4-4162-be3f-6cd9405ac466-metrics-certs") pod "network-metrics-daemon-mpgkn" (UID: "8aa144f2-e9b4-4162-be3f-6cd9405ac466") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:42.542081 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:42.542067 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096b3bbb-ead1-451f-9f3a-e5a4add161ad-original-pull-secret podName:096b3bbb-ead1-451f-9f3a-e5a4add161ad nodeName:}" failed. No retries permitted until 2026-08-04 17:52:58.542049535 +0000 UTC m=+34.232740579 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/096b3bbb-ead1-451f-9f3a-e5a4add161ad-original-pull-secret") pod "global-pull-secret-syncer-xbhvg" (UID: "096b3bbb-ead1-451f-9f3a-e5a4add161ad") : object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:42.642443 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:42.642408 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-479kg\" (UniqueName: \"kubernetes.io/projected/4bb95b54-5442-43b6-befa-8ecf7b8f4781-kube-api-access-479kg\") pod \"network-check-target-9z2cq\" (UID: \"4bb95b54-5442-43b6-befa-8ecf7b8f4781\") " pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:42.642626 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:42.642585 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 17:52:42.642626 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:42.642611 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 17:52:42.642626 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:42.642624 2572 projected.go:196] Error preparing data for projected volume kube-api-access-479kg for pod openshift-network-diagnostics/network-check-target-9z2cq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:42.642815 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:42.642699 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bb95b54-5442-43b6-befa-8ecf7b8f4781-kube-api-access-479kg podName:4bb95b54-5442-43b6-befa-8ecf7b8f4781 nodeName:}" failed. No retries permitted until 2026-08-04 17:52:58.642679402 +0000 UTC m=+34.333370445 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-479kg" (UniqueName: "kubernetes.io/projected/4bb95b54-5442-43b6-befa-8ecf7b8f4781-kube-api-access-479kg") pod "network-check-target-9z2cq" (UID: "4bb95b54-5442-43b6-befa-8ecf7b8f4781") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:42.816511 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:42.816422 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:42.816511 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:42.816471 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:42.817057 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:42.816576 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:52:42.817057 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:42.816734 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:52:43.815959 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:43.815876 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:43.816153 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:43.816007 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:52:44.817211 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:44.816698 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:44.817211 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:44.816845 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:52:44.817211 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:44.816934 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:44.817211 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:44.817102 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:52:45.815704 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:45.815654 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:45.815884 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:45.815802 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:52:46.816658 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:46.816623 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:46.817236 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:46.816684 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:46.817236 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:46.816787 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:52:46.817236 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:46.816942 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:52:47.816622 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:47.816280 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:47.816757 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:47.816654 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:52:47.992463 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:47.992434 2572 generic.go:356] "Generic (PLEG): container finished" podID="a1dbfaf6-9a2f-4146-807c-418181c922c1" containerID="7153186b0faae958a535c2e198f8c5d17c766700b782734b872342f712c3dd8d" exitCode=0 Aug 04 17:52:47.992600 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:47.992506 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2nnw" event={"ID":"a1dbfaf6-9a2f-4146-807c-418181c922c1","Type":"ContainerDied","Data":"7153186b0faae958a535c2e198f8c5d17c766700b782734b872342f712c3dd8d"} Aug 04 17:52:47.994332 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:47.994310 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6ns66" event={"ID":"8d005c33-5c1e-4ec8-990d-ee4d27743422","Type":"ContainerStarted","Data":"b4f186a7a96598cf70b84bb43d0daf58833394d21b519d0164f4cea8b755dfe3"} Aug 04 17:52:47.995990 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:47.995941 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" event={"ID":"3616edbe-d877-4980-b66a-e96223d78562","Type":"ContainerStarted","Data":"4d688a79ad7575b0a40daaba02873f0f43d7ff31e19309e4c03ece30115b14e8"} Aug 04 17:52:47.997391 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:47.997369 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4dgxz" event={"ID":"c600bdf9-3cef-457a-bb0b-704c5856c077","Type":"ContainerStarted","Data":"764f155a2352a978ed821f9ac63ce64b79fb91cf2320708ca3df5c8dcd110b7d"} Aug 04 17:52:48.000878 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:48.000854 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" event={"ID":"c11d0415-39ac-4682-81d2-130d039fae98","Type":"ContainerStarted","Data":"843d9a1f3a8c2f7d6789de05879e7280ed4cc802269a3af653e7ca924c9a9804"} Aug 04 17:52:48.000959 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:48.000931 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" event={"ID":"c11d0415-39ac-4682-81d2-130d039fae98","Type":"ContainerStarted","Data":"2bbf8cf285e4242a9ac5fd09b1504dc4667f4de538ff37b55081f1042ec707b2"} Aug 04 17:52:48.001014 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:48.000984 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" event={"ID":"c11d0415-39ac-4682-81d2-130d039fae98","Type":"ContainerStarted","Data":"94b1302cf63dc820c5941a6be3667084b9e3874e90940a10c83b3c4e0625f72f"} Aug 04 17:52:48.001049 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:48.001035 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" event={"ID":"c11d0415-39ac-4682-81d2-130d039fae98","Type":"ContainerStarted","Data":"65519b70b28514741c376e853751409b61c5368613b12721a15ed1c548eb56e7"} Aug 04 17:52:48.001108 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:48.001094 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" event={"ID":"c11d0415-39ac-4682-81d2-130d039fae98","Type":"ContainerStarted","Data":"ace97efc77a993106d68fdb11797237476bed6456ab41bc81d25082d2c179785"} Aug 04 17:52:48.001164 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:48.001147 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" event={"ID":"c11d0415-39ac-4682-81d2-130d039fae98","Type":"ContainerStarted","Data":"6713830da8a7a6aae5845e4fea5d20b75d3226aad2a836e4476dbbdefd8fc80a"} Aug 04 17:52:48.002252 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:48.002233 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-64c4r" event={"ID":"b3f65ebe-8341-406d-bc45-a6e176fccd97","Type":"ContainerStarted","Data":"f7132657f3fb421428fbdc18996bc85e2f47c3e91cbbc563d200d8f38d010fb3"} Aug 04 17:52:48.003814 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:48.003793 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qqgm5" event={"ID":"64a17319-0103-46f7-95fb-e5b9369434ad","Type":"ContainerStarted","Data":"16ff2f55625e1735cd690c143b6f230c187ebcc66d93854d8e6f67c8d74abbaf"} Aug 04 17:52:48.005340 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:48.005321 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" event={"ID":"610b5dc3-b664-4744-b311-85bbd40575b4","Type":"ContainerStarted","Data":"eaadbb1d2a8865bcd9b28b000e6ae4da80edae6a8fcdc49bcf042d3578e231a7"} Aug 04 17:52:48.068767 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:48.068679 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-64c4r" podStartSLOduration=7.649744891 podStartE2EDuration="23.068647185s" totalImagesPullingTime="15.418902294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:52:48.043060796 +0000 UTC m=+23.733751848" watchObservedRunningTime="2026-08-04 17:52:48.068647185 +0000 UTC m=+23.759338236" Aug 04 17:52:48.068892 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:48.068764 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dt2dw" podStartSLOduration=3.6949005059999998 podStartE2EDuration="23.068759917s" totalImagesPullingTime="19.373859411s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:52:48.066215055 +0000 UTC m=+23.756906119" watchObservedRunningTime="2026-08-04 17:52:48.068759917 +0000 UTC m=+23.759450967" Aug 04 17:52:48.082885 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:48.082745 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4dgxz" podStartSLOduration=3.725386993 podStartE2EDuration="23.082730916s" totalImagesPullingTime="19.357343923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:52:48.082007148 +0000 UTC m=+23.772698201" watchObservedRunningTime="2026-08-04 17:52:48.082730916 +0000 UTC m=+23.773421968" Aug 04 17:52:48.118715 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:48.118652 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6ns66" podStartSLOduration=3.774828892 podStartE2EDuration="23.118638691s" totalImagesPullingTime="19.343809799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:52:48.099134266 +0000 UTC m=+23.789825318" watchObservedRunningTime="2026-08-04 17:52:48.118638691 +0000 UTC m=+23.809329743" Aug 04 17:52:48.118834 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:48.118818 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qqgm5" podStartSLOduration=3.711240856 podStartE2EDuration="23.118810356s" totalImagesPullingTime="19.4075695s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:52:48.118560296 +0000 UTC m=+23.809251348" watchObservedRunningTime="2026-08-04 17:52:48.118810356 +0000 UTC m=+23.809501407" Aug 04 17:52:48.503699 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:48.503648 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-64c4r" Aug 04 17:52:48.504203 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:48.504182 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-64c4r" Aug 04 17:52:48.816071 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:48.815994 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:48.816327 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:48.815999 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:48.816327 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:48.816110 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:52:48.816327 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:48.816214 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:52:49.008371 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:49.008347 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-66v2k" event={"ID":"4fbeee23-8c17-49ea-b198-0b4d1d79a517","Type":"ContainerStarted","Data":"710bdb6963f0d88222d8c9a363c8d9343e8e29071bd3bdb21e02ed7d8c066e20"} Aug 04 17:52:49.009184 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:49.009161 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-64c4r" Aug 04 17:52:49.009271 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:49.009236 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-64c4r" Aug 04 17:52:49.025680 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:49.025609 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-66v2k" podStartSLOduration=4.680186433 podStartE2EDuration="24.025589294s" totalImagesPullingTime="19.345402861s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:52:49.024598096 +0000 UTC m=+24.715289174" watchObservedRunningTime="2026-08-04 17:52:49.025589294 +0000 UTC m=+24.716280346" Aug 04 17:52:49.069497 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:49.069445 2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 04 17:52:49.079507 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:49.079481 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:49.178156 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:49.178116 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:49.388065 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:49.387985 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:49.580248 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:49.580186 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:49.766449 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:49.766317 2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T17:52:49.069465826Z","UUID":"f6fb7fef-88f0-487d-aead-9cb0a78068b9","Handler":null,"Name":"","Endpoint":""} Aug 04 17:52:49.768032 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:49.768008 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 04 17:52:49.768180 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:49.768044 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 04 17:52:49.782653 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:49.782625 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:49.816171 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:49.816142 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:49.816332 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:49.816270 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:52:49.883893 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:49.883841 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:49.984868 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:49.984840 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:50.012872 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:50.012837 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" event={"ID":"610b5dc3-b664-4744-b311-85bbd40575b4","Type":"ContainerStarted","Data":"c3bac90da68ae3a80d7d5a8a1843fa9dc68fc0a1200fe84cc5d08494cd8470b9"} Aug 04 17:52:50.816558 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:50.816313 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:50.816719 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:50.816356 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:50.816719 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:50.816642 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:52:50.816841 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:50.816779 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:52:50.908103 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:50.908073 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:51.016204 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:51.016174 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:51.017738 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:51.017710 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" event={"ID":"610b5dc3-b664-4744-b311-85bbd40575b4","Type":"ContainerStarted","Data":"d63c35f0b120b1f6a1e2176a8aaf1d500d47bbe9c06f4115dcf1d27d09b72fa4"} Aug 04 17:52:51.021350 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:51.021313 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" event={"ID":"c11d0415-39ac-4682-81d2-130d039fae98","Type":"ContainerStarted","Data":"03b4b9aacaea943b603bbf5f94cf82b0f681943e03f3d94a966353f537f5dd08"} Aug 04 17:52:51.046711 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:51.046651 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57kll" podStartSLOduration=3.989817281 podStartE2EDuration="26.046637507s" totalImagesPullingTime="22.056820226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:52:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:52:51.046300996 +0000 UTC m=+26.736992049" watchObservedRunningTime="2026-08-04 17:52:51.046637507 +0000 UTC m=+26.737328554" Aug 04 17:52:51.169793 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:51.169705 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:51.402286 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:51.402256 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:51.585953 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:51.585881 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:51.781487 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:51.781444 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:51.815732 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:51.815701 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:51.815911 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:51.815824 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:52:51.883461 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:51.883382 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:52.179516 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:52.179489 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:52.816658 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:52.816483 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:52.816658 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:52.816487 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:52.817224 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:52.816807 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:52:52.817224 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:52.816824 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:52:53.006873 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:53.006788 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:53.026992 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:53.026966 2572 generic.go:356] "Generic (PLEG): container finished" podID="a1dbfaf6-9a2f-4146-807c-418181c922c1" containerID="a5bf82189f35b3fa05a5f50aa1c02f359bfc97e40c529383ef751741af7cdedd" exitCode=0 Aug 04 17:52:53.027125 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:53.027056 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2nnw" event={"ID":"a1dbfaf6-9a2f-4146-807c-418181c922c1","Type":"ContainerDied","Data":"a5bf82189f35b3fa05a5f50aa1c02f359bfc97e40c529383ef751741af7cdedd"} Aug 04 17:52:53.030858 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:53.030836 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" event={"ID":"c11d0415-39ac-4682-81d2-130d039fae98","Type":"ContainerStarted","Data":"59e3f0f686b9cf48cf7789f80545c99aaa833375bd1476019d0acae585ab669e"} Aug 04 17:52:53.031011 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:53.031000 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:53.031065 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:53.031036 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:53.046834 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:53.046812 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:53.079534 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:53.079500 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" podStartSLOduration=8.647936014999999 podStartE2EDuration="28.079487206s" totalImagesPullingTime="19.431551191s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:52:53.078990549 +0000 UTC m=+28.769681599" watchObservedRunningTime="2026-08-04 17:52:53.079487206 +0000 UTC m=+28.770178255" Aug 04 17:52:53.177855 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:53.177824 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:53.302503 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:53.302427 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:53.505319 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:53.505288 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:53.609300 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:53.609226 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:53.782852 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:53.782824 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:53.815835 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:53.815806 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:53.816014 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:53.815902 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:52:54.033647 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:54.033571 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:54.048504 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:54.048483 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:52:54.378976 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:54.378948 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:54.816575 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:54.816545 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:54.816778 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:54.816620 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:54.816778 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:54.816629 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:52:54.816778 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:54.816758 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:52:54.982909 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:54.982879 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:55.037325 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:55.037295 2572 generic.go:356] "Generic (PLEG): container finished" podID="a1dbfaf6-9a2f-4146-807c-418181c922c1" containerID="3712c1aa6891c78cc527d355ae97a143580167d9d94dd3ce9311a31010b63bb1" exitCode=0 Aug 04 17:52:55.037446 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:55.037364 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2nnw" event={"ID":"a1dbfaf6-9a2f-4146-807c-418181c922c1","Type":"ContainerDied","Data":"3712c1aa6891c78cc527d355ae97a143580167d9d94dd3ce9311a31010b63bb1"} Aug 04 17:52:55.176529 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:55.176417 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:55.378040 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:55.378000 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:55.578902 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:55.578825 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:55.816715 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:55.816657 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:55.816866 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:55.816788 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:52:56.139950 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:56.139921 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:56.239553 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:56.239518 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:56.589827 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:56.589798 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:56.788627 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:56.788596 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:56.816756 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:56.816726 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:56.816917 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:56.816826 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:52:56.816917 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:56.816888 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:56.817031 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:56.817004 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:52:57.044316 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:57.044242 2572 generic.go:356] "Generic (PLEG): container finished" podID="a1dbfaf6-9a2f-4146-807c-418181c922c1" containerID="05e7502f242f96cebad71287cedc78b81cd0a7ac2b8672c8ab2c753c6be530b5" exitCode=0 Aug 04 17:52:57.044316 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:57.044282 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2nnw" event={"ID":"a1dbfaf6-9a2f-4146-807c-418181c922c1","Type":"ContainerDied","Data":"05e7502f242f96cebad71287cedc78b81cd0a7ac2b8672c8ab2c753c6be530b5"} Aug 04 17:52:57.201165 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:57.201137 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:57.386942 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:57.386861 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:57.584659 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:57.584621 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:57.795322 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:57.795255 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:57.816302 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:57.816269 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:57.816446 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:57.816401 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:52:57.903282 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:57.903246 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:57.998536 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:57.998505 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:52:58.566806 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:58.566775 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/096b3bbb-ead1-451f-9f3a-e5a4add161ad-original-pull-secret\") pod \"global-pull-secret-syncer-xbhvg\" (UID: \"096b3bbb-ead1-451f-9f3a-e5a4add161ad\") " pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:58.566987 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:58.566823 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aa144f2-e9b4-4162-be3f-6cd9405ac466-metrics-certs\") pod \"network-metrics-daemon-mpgkn\" (UID: \"8aa144f2-e9b4-4162-be3f-6cd9405ac466\") " pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:58.566987 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:58.566936 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:58.566987 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:58.566963 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:58.567091 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:58.567009 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096b3bbb-ead1-451f-9f3a-e5a4add161ad-original-pull-secret podName:096b3bbb-ead1-451f-9f3a-e5a4add161ad nodeName:}" failed. No retries permitted until 2026-08-04 17:53:30.566991871 +0000 UTC m=+66.257682903 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/096b3bbb-ead1-451f-9f3a-e5a4add161ad-original-pull-secret") pod "global-pull-secret-syncer-xbhvg" (UID: "096b3bbb-ead1-451f-9f3a-e5a4add161ad") : object "kube-system"/"original-pull-secret" not registered Aug 04 17:52:58.567091 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:58.567025 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa144f2-e9b4-4162-be3f-6cd9405ac466-metrics-certs podName:8aa144f2-e9b4-4162-be3f-6cd9405ac466 nodeName:}" failed. No retries permitted until 2026-08-04 17:53:30.567018912 +0000 UTC m=+66.257709943 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8aa144f2-e9b4-4162-be3f-6cd9405ac466-metrics-certs") pod "network-metrics-daemon-mpgkn" (UID: "8aa144f2-e9b4-4162-be3f-6cd9405ac466") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 17:52:58.667639 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:58.667606 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-479kg\" (UniqueName: \"kubernetes.io/projected/4bb95b54-5442-43b6-befa-8ecf7b8f4781-kube-api-access-479kg\") pod \"network-check-target-9z2cq\" (UID: \"4bb95b54-5442-43b6-befa-8ecf7b8f4781\") " pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:58.667849 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:58.667764 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 17:52:58.667849 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:58.667783 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 17:52:58.667849 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:58.667793 2572 projected.go:196] Error preparing data for projected volume kube-api-access-479kg for pod openshift-network-diagnostics/network-check-target-9z2cq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:58.667849 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:58.667848 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bb95b54-5442-43b6-befa-8ecf7b8f4781-kube-api-access-479kg podName:4bb95b54-5442-43b6-befa-8ecf7b8f4781 nodeName:}" failed. No retries permitted until 2026-08-04 17:53:30.667831252 +0000 UTC m=+66.358522480 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-479kg" (UniqueName: "kubernetes.io/projected/4bb95b54-5442-43b6-befa-8ecf7b8f4781-kube-api-access-479kg") pod "network-check-target-9z2cq" (UID: "4bb95b54-5442-43b6-befa-8ecf7b8f4781") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 17:52:58.816541 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:58.816508 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:52:58.817036 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:58.816642 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:52:58.817036 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:58.816698 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:52:58.817036 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:58.816812 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:52:59.815852 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:52:59.815821 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:52:59.816003 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:52:59.815924 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:53:00.816849 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:00.816813 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:53:00.817301 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:00.816899 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:53:00.817301 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:00.817041 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:53:00.817301 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:00.817141 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:53:01.815930 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:01.815903 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:53:01.816087 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:01.816025 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:53:02.058923 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:02.058895 2572 generic.go:356] "Generic (PLEG): container finished" podID="a1dbfaf6-9a2f-4146-807c-418181c922c1" containerID="0e3a9c15c892d94ee9268dc2ea253f8658c6481d196165bd26c0378efc1f515c" exitCode=0 Aug 04 17:53:02.059290 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:02.058951 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2nnw" event={"ID":"a1dbfaf6-9a2f-4146-807c-418181c922c1","Type":"ContainerDied","Data":"0e3a9c15c892d94ee9268dc2ea253f8658c6481d196165bd26c0378efc1f515c"} Aug 04 17:53:02.816727 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:02.816528 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:53:02.816887 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:02.816569 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:53:02.816887 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:02.816842 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:53:02.817002 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:02.816880 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:53:03.063865 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:03.063836 2572 generic.go:356] "Generic (PLEG): container finished" podID="a1dbfaf6-9a2f-4146-807c-418181c922c1" containerID="9b2669450a049153935cde8a442f57acaa71bd2d12f3f86629c82b7661d1a037" exitCode=0 Aug 04 17:53:03.064181 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:03.063888 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2nnw" event={"ID":"a1dbfaf6-9a2f-4146-807c-418181c922c1","Type":"ContainerDied","Data":"9b2669450a049153935cde8a442f57acaa71bd2d12f3f86629c82b7661d1a037"} Aug 04 17:53:03.816538 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:03.816507 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:53:03.816715 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:03.816606 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:53:04.069803 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:04.069721 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t2nnw" event={"ID":"a1dbfaf6-9a2f-4146-807c-418181c922c1","Type":"ContainerStarted","Data":"d48a341b950af0f546dded1890b67d2547d48a175de9ab0ae820ab580f5c3bb3"} Aug 04 17:53:04.103479 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:04.103435 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t2nnw" podStartSLOduration=8.331815141 podStartE2EDuration="39.103420261s" totalImagesPullingTime="30.77160512s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:52:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:04.101765775 +0000 UTC m=+39.792456826" watchObservedRunningTime="2026-08-04 17:53:04.103420261 +0000 UTC m=+39.794111311" Aug 04 17:53:04.817167 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:04.817122 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:53:04.817341 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:04.817253 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:53:04.817341 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:04.817303 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:53:04.817455 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:04.817409 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:53:05.816385 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:05.816353 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:53:05.816768 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:05.816457 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:53:06.816504 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:06.816462 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:53:06.816921 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:06.816507 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:53:06.816921 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:06.816591 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:53:06.816921 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:06.816709 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:53:07.815857 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:07.815825 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:53:07.816031 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:07.815927 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:53:07.893341 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:07.893307 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:07.992874 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:07.992838 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:08.093206 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:08.093151 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:08.193353 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:08.193326 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:08.293258 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:08.293233 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:08.359461 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:08.359399 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:08.816184 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:08.816149 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:53:08.816378 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:08.816261 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:53:08.816378 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:08.816355 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:53:08.816500 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:08.816479 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:53:09.362957 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:09.362933 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:09.466140 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:09.466116 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:09.648266 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:09.648190 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9z2cq"] Aug 04 17:53:09.648390 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:09.648313 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:53:09.648447 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:09.648423 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:53:09.651245 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:09.651220 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xbhvg"] Aug 04 17:53:09.651383 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:09.651313 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:53:09.651441 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:09.651408 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:53:09.651885 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:09.651862 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mpgkn"] Aug 04 17:53:09.651986 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:09.651947 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:53:09.652067 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:09.652043 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:53:10.371677 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:10.371644 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:10.488012 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:10.487930 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:10.602249 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:10.602216 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:10.816557 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:10.816528 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:53:10.816719 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:10.816682 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:53:10.916379 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:10.916354 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:11.066960 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:11.066886 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:11.169443 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:11.169409 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:11.226829 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:11.226801 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:11.816018 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:11.815993 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:53:11.816395 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:11.815993 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:53:11.816395 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:11.816084 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:53:11.816395 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:11.816174 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:53:12.816529 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:12.816494 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:53:12.816997 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:12.816598 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpgkn" podUID="8aa144f2-e9b4-4162-be3f-6cd9405ac466" Aug 04 17:53:13.816337 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:13.816295 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:53:13.816532 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:13.816411 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xbhvg" podUID="096b3bbb-ead1-451f-9f3a-e5a4add161ad" Aug 04 17:53:13.816532 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:13.816469 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:53:13.816946 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:13.816579 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9z2cq" podUID="4bb95b54-5442-43b6-befa-8ecf7b8f4781" Aug 04 17:53:14.134911 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.134695 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeReady" Aug 04 17:53:14.135047 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.134967 2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 04 17:53:14.197576 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.197544 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wjdss"] Aug 04 17:53:14.214277 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.214254 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vxvb6"] Aug 04 17:53:14.214414 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.214377 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wjdss" Aug 04 17:53:14.217901 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.217838 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 04 17:53:14.217901 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.217861 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 04 17:53:14.218104 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.217943 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-z8v6x\"" Aug 04 17:53:14.218559 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.218541 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 04 17:53:14.230341 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.230321 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wjdss"] Aug 04 17:53:14.230439 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.230351 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vxvb6"] Aug 04 17:53:14.230439 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.230378 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bj5tc"] Aug 04 17:53:14.230537 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.230519 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vxvb6" Aug 04 17:53:14.233422 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.233401 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 04 17:53:14.233693 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.233656 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kwp6w\"" Aug 04 17:53:14.233798 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.233779 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 04 17:53:14.233798 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.233794 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 04 17:53:14.233915 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.233795 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 04 17:53:14.246406 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.246390 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bj5tc"] Aug 04 17:53:14.246481 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.246467 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bj5tc" Aug 04 17:53:14.249345 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.249327 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x4v5p\"" Aug 04 17:53:14.249450 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.249351 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 04 17:53:14.249549 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.249532 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 04 17:53:14.284865 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.284835 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqcwz\" (UniqueName: \"kubernetes.io/projected/ea1559b9-3848-4974-9c46-eac22ac768e3-kube-api-access-tqcwz\") pod \"insights-runtime-extractor-vxvb6\" (UID: \"ea1559b9-3848-4974-9c46-eac22ac768e3\") " pod="openshift-insights/insights-runtime-extractor-vxvb6" Aug 04 17:53:14.284865 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.284864 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6dad9eb4-a215-485c-961f-412ef770df62-config-volume\") pod \"dns-default-bj5tc\" (UID: \"6dad9eb4-a215-485c-961f-412ef770df62\") " pod="openshift-dns/dns-default-bj5tc" Aug 04 17:53:14.285084 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.284907 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz5dd\" (UniqueName: \"kubernetes.io/projected/3505d3f2-19bc-4965-ae4d-3dc9bff42667-kube-api-access-sz5dd\") pod \"ingress-canary-wjdss\" (UID: \"3505d3f2-19bc-4965-ae4d-3dc9bff42667\") " pod="openshift-ingress-canary/ingress-canary-wjdss" Aug 04 17:53:14.285084 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.284959 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea1559b9-3848-4974-9c46-eac22ac768e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vxvb6\" (UID: \"ea1559b9-3848-4974-9c46-eac22ac768e3\") " pod="openshift-insights/insights-runtime-extractor-vxvb6" Aug 04 17:53:14.285084 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.285017 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dad9eb4-a215-485c-961f-412ef770df62-tmp-dir\") pod \"dns-default-bj5tc\" (UID: \"6dad9eb4-a215-485c-961f-412ef770df62\") " pod="openshift-dns/dns-default-bj5tc" Aug 04 17:53:14.285084 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.285046 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pgsn\" (UniqueName: \"kubernetes.io/projected/6dad9eb4-a215-485c-961f-412ef770df62-kube-api-access-4pgsn\") pod \"dns-default-bj5tc\" (UID: \"6dad9eb4-a215-485c-961f-412ef770df62\") " pod="openshift-dns/dns-default-bj5tc" Aug 04 17:53:14.285250 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.285107 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ea1559b9-3848-4974-9c46-eac22ac768e3-crio-socket\") pod \"insights-runtime-extractor-vxvb6\" (UID: \"ea1559b9-3848-4974-9c46-eac22ac768e3\") " pod="openshift-insights/insights-runtime-extractor-vxvb6" Aug 04 17:53:14.285250 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.285173 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea1559b9-3848-4974-9c46-eac22ac768e3-data-volume\") pod \"insights-runtime-extractor-vxvb6\" (UID: \"ea1559b9-3848-4974-9c46-eac22ac768e3\") " pod="openshift-insights/insights-runtime-extractor-vxvb6" Aug 04 17:53:14.285250 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.285196 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea1559b9-3848-4974-9c46-eac22ac768e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vxvb6\" (UID: \"ea1559b9-3848-4974-9c46-eac22ac768e3\") " pod="openshift-insights/insights-runtime-extractor-vxvb6" Aug 04 17:53:14.285250 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.285225 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6dad9eb4-a215-485c-961f-412ef770df62-metrics-tls\") pod \"dns-default-bj5tc\" (UID: \"6dad9eb4-a215-485c-961f-412ef770df62\") " pod="openshift-dns/dns-default-bj5tc" Aug 04 17:53:14.285376 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.285251 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3505d3f2-19bc-4965-ae4d-3dc9bff42667-cert\") pod \"ingress-canary-wjdss\" (UID: \"3505d3f2-19bc-4965-ae4d-3dc9bff42667\") " pod="openshift-ingress-canary/ingress-canary-wjdss" Aug 04 17:53:14.386244 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.386137 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz5dd\" (UniqueName: \"kubernetes.io/projected/3505d3f2-19bc-4965-ae4d-3dc9bff42667-kube-api-access-sz5dd\") pod \"ingress-canary-wjdss\" (UID: \"3505d3f2-19bc-4965-ae4d-3dc9bff42667\") " pod="openshift-ingress-canary/ingress-canary-wjdss" Aug 04 17:53:14.386244 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.386171 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea1559b9-3848-4974-9c46-eac22ac768e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vxvb6\" (UID: \"ea1559b9-3848-4974-9c46-eac22ac768e3\") " pod="openshift-insights/insights-runtime-extractor-vxvb6" Aug 04 17:53:14.386244 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.386202 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dad9eb4-a215-485c-961f-412ef770df62-tmp-dir\") pod \"dns-default-bj5tc\" (UID: \"6dad9eb4-a215-485c-961f-412ef770df62\") " pod="openshift-dns/dns-default-bj5tc" Aug 04 17:53:14.386244 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.386221 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pgsn\" (UniqueName: \"kubernetes.io/projected/6dad9eb4-a215-485c-961f-412ef770df62-kube-api-access-4pgsn\") pod \"dns-default-bj5tc\" (UID: \"6dad9eb4-a215-485c-961f-412ef770df62\") " pod="openshift-dns/dns-default-bj5tc" Aug 04 17:53:14.386244 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.386248 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ea1559b9-3848-4974-9c46-eac22ac768e3-crio-socket\") pod \"insights-runtime-extractor-vxvb6\" (UID: \"ea1559b9-3848-4974-9c46-eac22ac768e3\") " pod="openshift-insights/insights-runtime-extractor-vxvb6" Aug 04 17:53:14.386550 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.386281 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea1559b9-3848-4974-9c46-eac22ac768e3-data-volume\") pod \"insights-runtime-extractor-vxvb6\" (UID: \"ea1559b9-3848-4974-9c46-eac22ac768e3\") " pod="openshift-insights/insights-runtime-extractor-vxvb6" Aug 04 17:53:14.386550 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.386298 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea1559b9-3848-4974-9c46-eac22ac768e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vxvb6\" (UID: \"ea1559b9-3848-4974-9c46-eac22ac768e3\") " pod="openshift-insights/insights-runtime-extractor-vxvb6" Aug 04 17:53:14.386550 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.386316 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6dad9eb4-a215-485c-961f-412ef770df62-metrics-tls\") pod \"dns-default-bj5tc\" (UID: \"6dad9eb4-a215-485c-961f-412ef770df62\") " pod="openshift-dns/dns-default-bj5tc" Aug 04 17:53:14.386550 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.386340 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3505d3f2-19bc-4965-ae4d-3dc9bff42667-cert\") pod \"ingress-canary-wjdss\" (UID: \"3505d3f2-19bc-4965-ae4d-3dc9bff42667\") " pod="openshift-ingress-canary/ingress-canary-wjdss" Aug 04 17:53:14.386550 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.386363 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqcwz\" (UniqueName: \"kubernetes.io/projected/ea1559b9-3848-4974-9c46-eac22ac768e3-kube-api-access-tqcwz\") pod \"insights-runtime-extractor-vxvb6\" (UID: \"ea1559b9-3848-4974-9c46-eac22ac768e3\") " pod="openshift-insights/insights-runtime-extractor-vxvb6" Aug 04 17:53:14.386550 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.386385 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6dad9eb4-a215-485c-961f-412ef770df62-config-volume\") pod \"dns-default-bj5tc\" (UID: \"6dad9eb4-a215-485c-961f-412ef770df62\") " pod="openshift-dns/dns-default-bj5tc" Aug 04 17:53:14.386550 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.386477 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ea1559b9-3848-4974-9c46-eac22ac768e3-crio-socket\") pod \"insights-runtime-extractor-vxvb6\" (UID: \"ea1559b9-3848-4974-9c46-eac22ac768e3\") " pod="openshift-insights/insights-runtime-extractor-vxvb6" Aug 04 17:53:14.386873 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.386561 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dad9eb4-a215-485c-961f-412ef770df62-tmp-dir\") pod \"dns-default-bj5tc\" (UID: \"6dad9eb4-a215-485c-961f-412ef770df62\") " pod="openshift-dns/dns-default-bj5tc" Aug 04 17:53:14.386873 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.386620 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea1559b9-3848-4974-9c46-eac22ac768e3-data-volume\") pod \"insights-runtime-extractor-vxvb6\" (UID: \"ea1559b9-3848-4974-9c46-eac22ac768e3\") " pod="openshift-insights/insights-runtime-extractor-vxvb6" Aug 04 17:53:14.387017 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.386996 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6dad9eb4-a215-485c-961f-412ef770df62-config-volume\") pod \"dns-default-bj5tc\" (UID: \"6dad9eb4-a215-485c-961f-412ef770df62\") " pod="openshift-dns/dns-default-bj5tc" Aug 04 17:53:14.390634 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.390611 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6dad9eb4-a215-485c-961f-412ef770df62-metrics-tls\") pod \"dns-default-bj5tc\" (UID: \"6dad9eb4-a215-485c-961f-412ef770df62\") " pod="openshift-dns/dns-default-bj5tc" Aug 04 17:53:14.390772 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.390612 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea1559b9-3848-4974-9c46-eac22ac768e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vxvb6\" (UID: \"ea1559b9-3848-4974-9c46-eac22ac768e3\") " pod="openshift-insights/insights-runtime-extractor-vxvb6" Aug 04 17:53:14.390772 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.390647 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3505d3f2-19bc-4965-ae4d-3dc9bff42667-cert\") pod \"ingress-canary-wjdss\" (UID: \"3505d3f2-19bc-4965-ae4d-3dc9bff42667\") " pod="openshift-ingress-canary/ingress-canary-wjdss" Aug 04 17:53:14.391052 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.391037 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:14.396472 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.396450 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea1559b9-3848-4974-9c46-eac22ac768e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vxvb6\" (UID: \"ea1559b9-3848-4974-9c46-eac22ac768e3\") " pod="openshift-insights/insights-runtime-extractor-vxvb6" Aug 04 17:53:14.396716 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.396699 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqcwz\" (UniqueName: \"kubernetes.io/projected/ea1559b9-3848-4974-9c46-eac22ac768e3-kube-api-access-tqcwz\") pod \"insights-runtime-extractor-vxvb6\" (UID: \"ea1559b9-3848-4974-9c46-eac22ac768e3\") " pod="openshift-insights/insights-runtime-extractor-vxvb6" Aug 04 17:53:14.396958 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.396928 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pgsn\" (UniqueName: \"kubernetes.io/projected/6dad9eb4-a215-485c-961f-412ef770df62-kube-api-access-4pgsn\") pod \"dns-default-bj5tc\" (UID: \"6dad9eb4-a215-485c-961f-412ef770df62\") " pod="openshift-dns/dns-default-bj5tc" Aug 04 17:53:14.397544 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.397521 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz5dd\" (UniqueName: \"kubernetes.io/projected/3505d3f2-19bc-4965-ae4d-3dc9bff42667-kube-api-access-sz5dd\") pod \"ingress-canary-wjdss\" (UID: \"3505d3f2-19bc-4965-ae4d-3dc9bff42667\") " pod="openshift-ingress-canary/ingress-canary-wjdss" Aug 04 17:53:14.527360 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.527324 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wjdss" Aug 04 17:53:14.538956 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.538936 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vxvb6" Aug 04 17:53:14.553903 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.553878 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bj5tc" Aug 04 17:53:14.749195 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.749152 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vxvb6"] Aug 04 17:53:14.750362 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.750339 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bj5tc"] Aug 04 17:53:14.753366 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:53:14.753299 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea1559b9_3848_4974_9c46_eac22ac768e3.slice/crio-df3df9c89be6f96821f82f952d9834b94407f3168c19e2113e60f2d16420fc4b WatchSource:0}: Error finding container df3df9c89be6f96821f82f952d9834b94407f3168c19e2113e60f2d16420fc4b: Status 404 returned error can't find the container with id df3df9c89be6f96821f82f952d9834b94407f3168c19e2113e60f2d16420fc4b Aug 04 17:53:14.754524 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:53:14.754498 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dad9eb4_a215_485c_961f_412ef770df62.slice/crio-d69665069aad59946f3c17bc77a0ec83c3e60c33fe5e6c5b503edea2e68f7bf2 WatchSource:0}: Error finding container d69665069aad59946f3c17bc77a0ec83c3e60c33fe5e6c5b503edea2e68f7bf2: Status 404 returned error can't find the container with id d69665069aad59946f3c17bc77a0ec83c3e60c33fe5e6c5b503edea2e68f7bf2 Aug 04 17:53:14.754633 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.754551 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wjdss"] Aug 04 17:53:14.757794 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:53:14.757773 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3505d3f2_19bc_4965_ae4d_3dc9bff42667.slice/crio-eb3b9b423529bed1bff5676dff8baea6fa1fe9d0e673a5987a9b7e65136390dd WatchSource:0}: Error finding container eb3b9b423529bed1bff5676dff8baea6fa1fe9d0e673a5987a9b7e65136390dd: Status 404 returned error can't find the container with id eb3b9b423529bed1bff5676dff8baea6fa1fe9d0e673a5987a9b7e65136390dd Aug 04 17:53:14.816555 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.816530 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:53:14.819368 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.819349 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-smv6k\"" Aug 04 17:53:14.820318 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:14.820299 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 17:53:15.095320 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.095121 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wjdss" event={"ID":"3505d3f2-19bc-4965-ae4d-3dc9bff42667","Type":"ContainerStarted","Data":"eb3b9b423529bed1bff5676dff8baea6fa1fe9d0e673a5987a9b7e65136390dd"} Aug 04 17:53:15.096279 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.096243 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bj5tc" event={"ID":"6dad9eb4-a215-485c-961f-412ef770df62","Type":"ContainerStarted","Data":"d69665069aad59946f3c17bc77a0ec83c3e60c33fe5e6c5b503edea2e68f7bf2"} Aug 04 17:53:15.097620 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.097600 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vxvb6" event={"ID":"ea1559b9-3848-4974-9c46-eac22ac768e3","Type":"ContainerStarted","Data":"7414d5420e3184c9f2b9fc085ca92ad1bc1d5c83003cebbddd57452cd6a36e9d"} Aug 04 17:53:15.097730 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.097641 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vxvb6" event={"ID":"ea1559b9-3848-4974-9c46-eac22ac768e3","Type":"ContainerStarted","Data":"df3df9c89be6f96821f82f952d9834b94407f3168c19e2113e60f2d16420fc4b"} Aug 04 17:53:15.660459 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.660428 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:15.757211 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.757184 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:15.774710 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.774689 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-hbqtm"] Aug 04 17:53:15.776689 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.776654 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-hbqtm" Aug 04 17:53:15.780283 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.780266 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 04 17:53:15.780877 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.780724 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Aug 04 17:53:15.781379 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.781357 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Aug 04 17:53:15.781764 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.781572 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 04 17:53:15.783908 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.783888 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 04 17:53:15.786930 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.786912 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8pj87\"" Aug 04 17:53:15.793424 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.793403 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-hbqtm"] Aug 04 17:53:15.816087 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.816060 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:53:15.816480 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.816457 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:53:15.821613 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.821592 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 17:53:15.822057 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.821649 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-f7jt5\"" Aug 04 17:53:15.822309 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.822291 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 17:53:15.822545 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.822525 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 17:53:15.897285 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.897248 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ef17428-3a91-41ab-b3a3-525d6eaffa6f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-hbqtm\" (UID: \"5ef17428-3a91-41ab-b3a3-525d6eaffa6f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hbqtm" Aug 04 17:53:15.897473 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.897300 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r56wc\" (UniqueName: \"kubernetes.io/projected/5ef17428-3a91-41ab-b3a3-525d6eaffa6f-kube-api-access-r56wc\") pod \"prometheus-operator-758d7cb874-hbqtm\" (UID: \"5ef17428-3a91-41ab-b3a3-525d6eaffa6f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hbqtm" Aug 04 17:53:15.897473 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.897392 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ef17428-3a91-41ab-b3a3-525d6eaffa6f-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-hbqtm\" (UID: \"5ef17428-3a91-41ab-b3a3-525d6eaffa6f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hbqtm" Aug 04 17:53:15.897473 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.897453 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ef17428-3a91-41ab-b3a3-525d6eaffa6f-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-hbqtm\" (UID: \"5ef17428-3a91-41ab-b3a3-525d6eaffa6f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hbqtm" Aug 04 17:53:15.999041 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.998469 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ef17428-3a91-41ab-b3a3-525d6eaffa6f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-hbqtm\" (UID: \"5ef17428-3a91-41ab-b3a3-525d6eaffa6f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hbqtm" Aug 04 17:53:15.999041 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.998534 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r56wc\" (UniqueName: \"kubernetes.io/projected/5ef17428-3a91-41ab-b3a3-525d6eaffa6f-kube-api-access-r56wc\") pod \"prometheus-operator-758d7cb874-hbqtm\" (UID: \"5ef17428-3a91-41ab-b3a3-525d6eaffa6f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hbqtm" Aug 04 17:53:15.999041 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.998608 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ef17428-3a91-41ab-b3a3-525d6eaffa6f-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-hbqtm\" (UID: \"5ef17428-3a91-41ab-b3a3-525d6eaffa6f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hbqtm" Aug 04 17:53:15.999041 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.998654 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ef17428-3a91-41ab-b3a3-525d6eaffa6f-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-hbqtm\" (UID: \"5ef17428-3a91-41ab-b3a3-525d6eaffa6f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hbqtm" Aug 04 17:53:15.999041 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:15.998782 2572 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Aug 04 17:53:15.999041 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:15.998842 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ef17428-3a91-41ab-b3a3-525d6eaffa6f-prometheus-operator-tls podName:5ef17428-3a91-41ab-b3a3-525d6eaffa6f nodeName:}" failed. No retries permitted until 2026-08-04 17:53:16.498823109 +0000 UTC m=+52.189514141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/5ef17428-3a91-41ab-b3a3-525d6eaffa6f-prometheus-operator-tls") pod "prometheus-operator-758d7cb874-hbqtm" (UID: "5ef17428-3a91-41ab-b3a3-525d6eaffa6f") : secret "prometheus-operator-tls" not found Aug 04 17:53:15.999509 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:15.999489 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ef17428-3a91-41ab-b3a3-525d6eaffa6f-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-hbqtm\" (UID: \"5ef17428-3a91-41ab-b3a3-525d6eaffa6f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hbqtm" Aug 04 17:53:16.003262 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:16.003238 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ef17428-3a91-41ab-b3a3-525d6eaffa6f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-hbqtm\" (UID: \"5ef17428-3a91-41ab-b3a3-525d6eaffa6f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hbqtm" Aug 04 17:53:16.012006 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:16.011986 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r56wc\" (UniqueName: \"kubernetes.io/projected/5ef17428-3a91-41ab-b3a3-525d6eaffa6f-kube-api-access-r56wc\") pod \"prometheus-operator-758d7cb874-hbqtm\" (UID: \"5ef17428-3a91-41ab-b3a3-525d6eaffa6f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hbqtm" Aug 04 17:53:16.103135 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:16.103105 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vxvb6" event={"ID":"ea1559b9-3848-4974-9c46-eac22ac768e3","Type":"ContainerStarted","Data":"8c8e3d4b2c1ee5b8d184536ad9d5944e8825d37b9f7bddd857ff19ead8bb452a"} Aug 04 17:53:16.437297 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:16.437253 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:16.502512 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:16.502469 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ef17428-3a91-41ab-b3a3-525d6eaffa6f-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-hbqtm\" (UID: \"5ef17428-3a91-41ab-b3a3-525d6eaffa6f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hbqtm" Aug 04 17:53:16.505248 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:16.505224 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ef17428-3a91-41ab-b3a3-525d6eaffa6f-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-hbqtm\" (UID: \"5ef17428-3a91-41ab-b3a3-525d6eaffa6f\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-hbqtm" Aug 04 17:53:16.687389 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:16.687335 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-hbqtm" Aug 04 17:53:16.893324 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:16.893288 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-hbqtm"] Aug 04 17:53:16.898876 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:53:16.898837 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef17428_3a91_41ab_b3a3_525d6eaffa6f.slice/crio-b52d0af2bf8109b567d9a555fbc2c25c877c506bc212a52c4b17f346f263a977 WatchSource:0}: Error finding container b52d0af2bf8109b567d9a555fbc2c25c877c506bc212a52c4b17f346f263a977: Status 404 returned error can't find the container with id b52d0af2bf8109b567d9a555fbc2c25c877c506bc212a52c4b17f346f263a977 Aug 04 17:53:17.107446 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:17.107413 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-hbqtm" event={"ID":"5ef17428-3a91-41ab-b3a3-525d6eaffa6f","Type":"ContainerStarted","Data":"b52d0af2bf8109b567d9a555fbc2c25c877c506bc212a52c4b17f346f263a977"} Aug 04 17:53:17.109887 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:17.109853 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bj5tc" event={"ID":"6dad9eb4-a215-485c-961f-412ef770df62","Type":"ContainerStarted","Data":"9e2a8db46afed612f51acf97d4d5844b867e87f8217b158bcdd98aa3880e39d2"} Aug 04 17:53:17.110012 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:17.109907 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bj5tc" event={"ID":"6dad9eb4-a215-485c-961f-412ef770df62","Type":"ContainerStarted","Data":"4cab82b0080b8765aac1e195833bf46b3fc7ca30a7d468e724e820b6eedef749"} Aug 04 17:53:17.110073 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:17.110017 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bj5tc" Aug 04 17:53:17.146444 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:17.146391 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bj5tc" podStartSLOduration=1.219252981 podStartE2EDuration="3.146371314s" totalImagesPullingTime="1.927118333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:53:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:17.145862048 +0000 UTC m=+52.836553096" watchObservedRunningTime="2026-08-04 17:53:17.146371314 +0000 UTC m=+52.837062369" Aug 04 17:53:17.390599 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:17.390528 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-77c974ffcf-gf5hr"] Aug 04 17:53:17.392567 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:17.392546 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-gf5hr" Aug 04 17:53:17.395245 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:17.395228 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-qkwzr\"" Aug 04 17:53:17.395328 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:17.395268 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Aug 04 17:53:17.395975 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:17.395960 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Aug 04 17:53:17.402686 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:17.402646 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:17.409851 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:17.409835 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-gf5hr"] Aug 04 17:53:17.496741 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:17.496707 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:17.513797 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:17.513769 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l4mg\" (UniqueName: \"kubernetes.io/projected/48ff3a32-3414-4512-a8d8-7918e3ae955a-kube-api-access-5l4mg\") pod \"downloads-77c974ffcf-gf5hr\" (UID: \"48ff3a32-3414-4512-a8d8-7918e3ae955a\") " pod="openshift-console/downloads-77c974ffcf-gf5hr" Aug 04 17:53:17.615139 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:17.615107 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l4mg\" (UniqueName: \"kubernetes.io/projected/48ff3a32-3414-4512-a8d8-7918e3ae955a-kube-api-access-5l4mg\") pod \"downloads-77c974ffcf-gf5hr\" (UID: \"48ff3a32-3414-4512-a8d8-7918e3ae955a\") " pod="openshift-console/downloads-77c974ffcf-gf5hr" Aug 04 17:53:17.626812 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:17.626778 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l4mg\" (UniqueName: \"kubernetes.io/projected/48ff3a32-3414-4512-a8d8-7918e3ae955a-kube-api-access-5l4mg\") pod \"downloads-77c974ffcf-gf5hr\" (UID: \"48ff3a32-3414-4512-a8d8-7918e3ae955a\") " pod="openshift-console/downloads-77c974ffcf-gf5hr" Aug 04 17:53:17.671575 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:17.671497 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:17.703507 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:17.703464 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-gf5hr" Aug 04 17:53:17.849514 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:17.849472 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:17.912485 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:17.912446 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:18.049971 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:18.049934 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:18.110726 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:18.110685 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:18.315835 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:18.315283 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-gf5hr"] Aug 04 17:53:18.322817 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:53:18.322791 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ff3a32_3414_4512_a8d8_7918e3ae955a.slice/crio-8365fa04013fa841ac478aa7f16b6f7c7799e6a6bb2393c8dddecdffc54e2c6b WatchSource:0}: Error finding container 8365fa04013fa841ac478aa7f16b6f7c7799e6a6bb2393c8dddecdffc54e2c6b: Status 404 returned error can't find the container with id 8365fa04013fa841ac478aa7f16b6f7c7799e6a6bb2393c8dddecdffc54e2c6b Aug 04 17:53:18.457291 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:18.457256 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:19.065614 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:19.065583 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:19.117459 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:19.117422 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wjdss" event={"ID":"3505d3f2-19bc-4965-ae4d-3dc9bff42667","Type":"ContainerStarted","Data":"855e5068b16ba3be4c3e3c9369c7631bb0ab8d8f37940b98c9f09939676becf5"} Aug 04 17:53:19.119380 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:19.119353 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-gf5hr" event={"ID":"48ff3a32-3414-4512-a8d8-7918e3ae955a","Type":"ContainerStarted","Data":"8365fa04013fa841ac478aa7f16b6f7c7799e6a6bb2393c8dddecdffc54e2c6b"} Aug 04 17:53:19.123247 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:19.122975 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vxvb6" event={"ID":"ea1559b9-3848-4974-9c46-eac22ac768e3","Type":"ContainerStarted","Data":"35fafb95fce75985af9c35138b15c01458f9c46f343eb91b8d454e97df441f3b"} Aug 04 17:53:19.134846 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:19.134804 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wjdss" podStartSLOduration=1.701385754 podStartE2EDuration="5.134785668s" totalImagesPullingTime="3.433399914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:53:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:19.134290763 +0000 UTC m=+54.824981813" watchObservedRunningTime="2026-08-04 17:53:19.134785668 +0000 UTC m=+54.825476719" Aug 04 17:53:19.169846 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:19.169808 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:19.269094 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:19.269066 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:19.404080 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:19.403999 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:20.128748 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:20.128711 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-hbqtm" event={"ID":"5ef17428-3a91-41ab-b3a3-525d6eaffa6f","Type":"ContainerStarted","Data":"25b9a3b08dfed9966299f3ac875aea0de01a67ec4da9824b434a1f4f98d2dcdc"} Aug 04 17:53:20.129581 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:20.128776 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-hbqtm" event={"ID":"5ef17428-3a91-41ab-b3a3-525d6eaffa6f","Type":"ContainerStarted","Data":"e9f1c9dba617c021a2fb279bff03fb3d828095c4a20d9c750133a6e1d508c6d7"} Aug 04 17:53:20.147609 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:20.147541 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vxvb6" podStartSLOduration=2.871927249 podStartE2EDuration="6.147518986s" totalImagesPullingTime="3.275591737s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:53:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:19.155103033 +0000 UTC m=+54.845794085" watchObservedRunningTime="2026-08-04 17:53:20.147518986 +0000 UTC m=+55.838210040" Aug 04 17:53:20.147964 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:20.147865 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-758d7cb874-hbqtm" podStartSLOduration=2.967712048 podStartE2EDuration="5.147850357s" totalImagesPullingTime="2.180138309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:53:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:20.146644306 +0000 UTC m=+55.837335356" watchObservedRunningTime="2026-08-04 17:53:20.147850357 +0000 UTC m=+55.838541407" Aug 04 17:53:20.279764 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:20.279731 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:20.727184 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:20.727139 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:20.814416 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:20.814376 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:20.886058 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:20.886026 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:21.907438 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:21.907401 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:22.151151 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.151099 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:22.199437 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.199366 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hnrz6"] Aug 04 17:53:22.228968 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.228933 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.231736 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.231714 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7kr7n\"" Aug 04 17:53:22.232381 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.232361 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 04 17:53:22.232494 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.232416 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 04 17:53:22.232722 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.232707 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 04 17:53:22.239078 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.239058 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:22.352284 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.352252 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.352442 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.352309 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-sys\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.352442 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.352352 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zdpl\" (UniqueName: \"kubernetes.io/projected/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-kube-api-access-6zdpl\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.352442 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.352408 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-root\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.352564 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.352473 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-node-exporter-textfile\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.352564 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.352518 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-node-exporter-accelerators-collector-config\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.352564 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.352558 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-metrics-client-ca\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.352736 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.352582 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-node-exporter-tls\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.352736 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.352657 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-node-exporter-wtmp\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.407873 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.407830 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:22.453480 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.453400 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.453480 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.453460 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-sys\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.453705 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.453491 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zdpl\" (UniqueName: \"kubernetes.io/projected/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-kube-api-access-6zdpl\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.453705 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.453519 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-root\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.453705 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.453600 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-node-exporter-textfile\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.453705 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.453648 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-node-exporter-accelerators-collector-config\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.453705 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.453697 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-metrics-client-ca\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.453978 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.453607 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-sys\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.453978 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.453726 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-node-exporter-tls\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.453978 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.453797 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-node-exporter-wtmp\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.453978 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:22.453812 2572 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Aug 04 17:53:22.453978 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:22.453865 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-node-exporter-tls podName:cfca7ce7-765d-4d8a-8ea1-a8f9c5931466 nodeName:}" failed. No retries permitted until 2026-08-04 17:53:22.953846139 +0000 UTC m=+58.644537183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-node-exporter-tls") pod "node-exporter-hnrz6" (UID: "cfca7ce7-765d-4d8a-8ea1-a8f9c5931466") : secret "node-exporter-tls" not found Aug 04 17:53:22.453978 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.453947 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-node-exporter-wtmp\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.453978 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.453607 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-root\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.454506 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.454455 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-node-exporter-textfile\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.454873 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.454846 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-metrics-client-ca\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.454975 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.454846 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-node-exporter-accelerators-collector-config\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.457840 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.457820 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.469496 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.469468 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zdpl\" (UniqueName: \"kubernetes.io/projected/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-kube-api-access-6zdpl\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.523236 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.523211 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:22.624495 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.624454 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:22.724409 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.724338 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:22.832916 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.832879 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:22.956618 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.956585 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-node-exporter-tls\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:22.959379 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:22.959346 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cfca7ce7-765d-4d8a-8ea1-a8f9c5931466-node-exporter-tls\") pod \"node-exporter-hnrz6\" (UID: \"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466\") " pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:23.141178 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.141147 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hnrz6" Aug 04 17:53:23.151909 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:53:23.151878 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfca7ce7_765d_4d8a_8ea1_a8f9c5931466.slice/crio-a180e39c8a7bfcddf4e43386787d247c42cda4c5110555772da837f28ba8d57e WatchSource:0}: Error finding container a180e39c8a7bfcddf4e43386787d247c42cda4c5110555772da837f28ba8d57e: Status 404 returned error can't find the container with id a180e39c8a7bfcddf4e43386787d247c42cda4c5110555772da837f28ba8d57e Aug 04 17:53:23.257531 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.257487 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:23.360459 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.360421 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:23.375752 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.375719 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 17:53:23.378707 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.378688 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.381772 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.381753 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Aug 04 17:53:23.382781 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.382758 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Aug 04 17:53:23.382885 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.382783 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Aug 04 17:53:23.383815 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.383790 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Aug 04 17:53:23.383914 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.383849 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Aug 04 17:53:23.383914 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.383894 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-c5x77\"" Aug 04 17:53:23.384104 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.384089 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Aug 04 17:53:23.384104 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.384097 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Aug 04 17:53:23.384218 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.384157 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Aug 04 17:53:23.384218 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.384194 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Aug 04 17:53:23.398528 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.398473 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 17:53:23.460979 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.460948 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.460979 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.460987 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.461225 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.461019 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.461225 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.461074 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.461225 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.461094 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.461225 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.461117 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6shv\" (UniqueName: \"kubernetes.io/projected/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-kube-api-access-n6shv\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.461225 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.461152 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.461225 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.461172 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-web-config\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.461225 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.461191 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.461225 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.461223 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-config-volume\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.461540 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.461242 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-config-out\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.461540 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.461265 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.461540 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.461294 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.562031 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.561997 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.562216 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.562040 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.562216 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.562068 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6shv\" (UniqueName: \"kubernetes.io/projected/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-kube-api-access-n6shv\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.562216 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.562096 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.562216 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.562123 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-web-config\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.562216 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.562145 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.562216 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.562194 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-config-volume\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.562216 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.562215 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-config-out\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.562555 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.562237 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.562555 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.562266 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.562555 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.562314 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.562555 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.562350 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.562555 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.562381 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.563510 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.563484 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.563993 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.563967 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.564325 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.564297 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.564789 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:23.564768 2572 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Aug 04 17:53:23.566118 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:53:23.564840 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-main-tls podName:15a69893-7efd-4ce6-98da-d6bbe0a2ec3b nodeName:}" failed. No retries permitted until 2026-08-04 17:53:24.064818844 +0000 UTC m=+59.755509887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b") : secret "alertmanager-main-tls" not found Aug 04 17:53:23.566118 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.565443 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-web-config\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.566118 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.565544 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.566118 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.565879 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.566305 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.566131 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-config-volume\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.567458 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.567420 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.567893 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.567861 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.568696 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.568650 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-config-out\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.569113 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.569089 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.574574 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.574537 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6shv\" (UniqueName: \"kubernetes.io/projected/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-kube-api-access-n6shv\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:23.897360 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:23.897327 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:24.067782 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:24.067746 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:24.070774 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:24.070750 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:24.144370 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:24.144333 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hnrz6" event={"ID":"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466","Type":"ContainerStarted","Data":"a180e39c8a7bfcddf4e43386787d247c42cda4c5110555772da837f28ba8d57e"} Aug 04 17:53:24.289311 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:24.289284 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:53:24.467280 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:24.467251 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 17:53:24.469823 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:53:24.469791 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a69893_7efd_4ce6_98da_d6bbe0a2ec3b.slice/crio-7da4d850e43744928fb4a64fcdd640088a9d884ec2a4c231064e226c572660f8 WatchSource:0}: Error finding container 7da4d850e43744928fb4a64fcdd640088a9d884ec2a4c231064e226c572660f8: Status 404 returned error can't find the container with id 7da4d850e43744928fb4a64fcdd640088a9d884ec2a4c231064e226c572660f8 Aug 04 17:53:25.150183 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.150140 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b","Type":"ContainerStarted","Data":"7da4d850e43744928fb4a64fcdd640088a9d884ec2a4c231064e226c572660f8"} Aug 04 17:53:25.153317 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.153286 2572 generic.go:356] "Generic (PLEG): container finished" podID="cfca7ce7-765d-4d8a-8ea1-a8f9c5931466" containerID="3a093a07718698c4ff1e42ed7b18c4ebab1381bdb87f4995af48b85e57ce5c13" exitCode=0 Aug 04 17:53:25.153459 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.153328 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hnrz6" event={"ID":"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466","Type":"ContainerDied","Data":"3a093a07718698c4ff1e42ed7b18c4ebab1381bdb87f4995af48b85e57ce5c13"} Aug 04 17:53:25.260207 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.260176 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:25.359917 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.359876 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:25.360684 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.360635 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w"] Aug 04 17:53:25.363730 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.363708 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.367759 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.366941 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Aug 04 17:53:25.367759 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.367154 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dhcp9lds7dglb\"" Aug 04 17:53:25.367911 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.367839 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Aug 04 17:53:25.367911 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.367885 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Aug 04 17:53:25.368015 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.367943 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Aug 04 17:53:25.368069 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.368030 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-cpbcw\"" Aug 04 17:53:25.368220 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.368200 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Aug 04 17:53:25.385212 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.385171 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w"] Aug 04 17:53:25.477693 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.477586 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2e3e902-9da0-498d-87ff-3e9040b86081-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c8d9fcf-f6b4w\" (UID: \"d2e3e902-9da0-498d-87ff-3e9040b86081\") " pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.477693 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.477641 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2e3e902-9da0-498d-87ff-3e9040b86081-metrics-client-ca\") pod \"thanos-querier-5c8d9fcf-f6b4w\" (UID: \"d2e3e902-9da0-498d-87ff-3e9040b86081\") " pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.477924 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.477698 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2e3e902-9da0-498d-87ff-3e9040b86081-secret-grpc-tls\") pod \"thanos-querier-5c8d9fcf-f6b4w\" (UID: \"d2e3e902-9da0-498d-87ff-3e9040b86081\") " pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.477924 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.477742 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d2e3e902-9da0-498d-87ff-3e9040b86081-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c8d9fcf-f6b4w\" (UID: \"d2e3e902-9da0-498d-87ff-3e9040b86081\") " pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.477924 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.477783 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2e3e902-9da0-498d-87ff-3e9040b86081-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c8d9fcf-f6b4w\" (UID: \"d2e3e902-9da0-498d-87ff-3e9040b86081\") " pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.477924 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.477811 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d2e3e902-9da0-498d-87ff-3e9040b86081-secret-thanos-querier-tls\") pod \"thanos-querier-5c8d9fcf-f6b4w\" (UID: \"d2e3e902-9da0-498d-87ff-3e9040b86081\") " pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.477924 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.477859 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d2e3e902-9da0-498d-87ff-3e9040b86081-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c8d9fcf-f6b4w\" (UID: \"d2e3e902-9da0-498d-87ff-3e9040b86081\") " pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.477924 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.477912 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpxdj\" (UniqueName: \"kubernetes.io/projected/d2e3e902-9da0-498d-87ff-3e9040b86081-kube-api-access-rpxdj\") pod \"thanos-querier-5c8d9fcf-f6b4w\" (UID: \"d2e3e902-9da0-498d-87ff-3e9040b86081\") " pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.579950 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.579911 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpxdj\" (UniqueName: \"kubernetes.io/projected/d2e3e902-9da0-498d-87ff-3e9040b86081-kube-api-access-rpxdj\") pod \"thanos-querier-5c8d9fcf-f6b4w\" (UID: \"d2e3e902-9da0-498d-87ff-3e9040b86081\") " pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.580135 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.579985 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2e3e902-9da0-498d-87ff-3e9040b86081-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c8d9fcf-f6b4w\" (UID: \"d2e3e902-9da0-498d-87ff-3e9040b86081\") " pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.580135 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.580017 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2e3e902-9da0-498d-87ff-3e9040b86081-metrics-client-ca\") pod \"thanos-querier-5c8d9fcf-f6b4w\" (UID: \"d2e3e902-9da0-498d-87ff-3e9040b86081\") " pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.580135 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.580046 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2e3e902-9da0-498d-87ff-3e9040b86081-secret-grpc-tls\") pod \"thanos-querier-5c8d9fcf-f6b4w\" (UID: \"d2e3e902-9da0-498d-87ff-3e9040b86081\") " pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.580295 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.580244 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d2e3e902-9da0-498d-87ff-3e9040b86081-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c8d9fcf-f6b4w\" (UID: \"d2e3e902-9da0-498d-87ff-3e9040b86081\") " pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.580362 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.580314 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2e3e902-9da0-498d-87ff-3e9040b86081-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c8d9fcf-f6b4w\" (UID: \"d2e3e902-9da0-498d-87ff-3e9040b86081\") " pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.580362 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.580348 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d2e3e902-9da0-498d-87ff-3e9040b86081-secret-thanos-querier-tls\") pod \"thanos-querier-5c8d9fcf-f6b4w\" (UID: \"d2e3e902-9da0-498d-87ff-3e9040b86081\") " pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.580457 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.580417 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d2e3e902-9da0-498d-87ff-3e9040b86081-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c8d9fcf-f6b4w\" (UID: \"d2e3e902-9da0-498d-87ff-3e9040b86081\") " pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.580857 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.580836 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2e3e902-9da0-498d-87ff-3e9040b86081-metrics-client-ca\") pod \"thanos-querier-5c8d9fcf-f6b4w\" (UID: \"d2e3e902-9da0-498d-87ff-3e9040b86081\") " pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.582644 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.582616 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2e3e902-9da0-498d-87ff-3e9040b86081-secret-grpc-tls\") pod \"thanos-querier-5c8d9fcf-f6b4w\" (UID: \"d2e3e902-9da0-498d-87ff-3e9040b86081\") " pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.583156 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.583130 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d2e3e902-9da0-498d-87ff-3e9040b86081-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c8d9fcf-f6b4w\" (UID: \"d2e3e902-9da0-498d-87ff-3e9040b86081\") " pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.583255 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.583132 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2e3e902-9da0-498d-87ff-3e9040b86081-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c8d9fcf-f6b4w\" (UID: \"d2e3e902-9da0-498d-87ff-3e9040b86081\") " pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.583316 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.583268 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2e3e902-9da0-498d-87ff-3e9040b86081-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c8d9fcf-f6b4w\" (UID: \"d2e3e902-9da0-498d-87ff-3e9040b86081\") " pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.583558 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.583541 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d2e3e902-9da0-498d-87ff-3e9040b86081-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c8d9fcf-f6b4w\" (UID: \"d2e3e902-9da0-498d-87ff-3e9040b86081\") " pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.583593 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.583580 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d2e3e902-9da0-498d-87ff-3e9040b86081-secret-thanos-querier-tls\") pod \"thanos-querier-5c8d9fcf-f6b4w\" (UID: \"d2e3e902-9da0-498d-87ff-3e9040b86081\") " pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.590449 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.590424 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpxdj\" (UniqueName: \"kubernetes.io/projected/d2e3e902-9da0-498d-87ff-3e9040b86081-kube-api-access-rpxdj\") pod \"thanos-querier-5c8d9fcf-f6b4w\" (UID: \"d2e3e902-9da0-498d-87ff-3e9040b86081\") " pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.678497 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.678470 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:25.761244 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.761197 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:25.828297 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.828267 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w"] Aug 04 17:53:25.830231 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:53:25.830200 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e3e902_9da0_498d_87ff_3e9040b86081.slice/crio-9adcbac1e0a1e4b4259173622eb5993f7dc04b34f7cae3c49b65f6db4bcf78ef WatchSource:0}: Error finding container 9adcbac1e0a1e4b4259173622eb5993f7dc04b34f7cae3c49b65f6db4bcf78ef: Status 404 returned error can't find the container with id 9adcbac1e0a1e4b4259173622eb5993f7dc04b34f7cae3c49b65f6db4bcf78ef Aug 04 17:53:25.869142 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:25.869101 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:26.052068 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:26.052042 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2lkw4" Aug 04 17:53:26.158549 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:26.158514 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" event={"ID":"d2e3e902-9da0-498d-87ff-3e9040b86081","Type":"ContainerStarted","Data":"9adcbac1e0a1e4b4259173622eb5993f7dc04b34f7cae3c49b65f6db4bcf78ef"} Aug 04 17:53:26.160373 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:26.160341 2572 generic.go:356] "Generic (PLEG): container finished" podID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerID="30247426d51c290e26690504fa14b92ffd999777a3a6f515efe362698500260c" exitCode=0 Aug 04 17:53:26.160532 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:26.160436 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b","Type":"ContainerDied","Data":"30247426d51c290e26690504fa14b92ffd999777a3a6f515efe362698500260c"} Aug 04 17:53:26.162797 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:26.162777 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hnrz6" event={"ID":"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466","Type":"ContainerStarted","Data":"55888dfc9ccb47cf3f8803dd3785c32ba921c35c5cbae603d88dfe0c0d8bae3d"} Aug 04 17:53:26.162915 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:26.162817 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hnrz6" event={"ID":"cfca7ce7-765d-4d8a-8ea1-a8f9c5931466","Type":"ContainerStarted","Data":"3cee3e3c85708abe8b410a1f3ed853a83868540849cb905fc8ce5dd1ea44fd5c"} Aug 04 17:53:26.211449 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:26.211405 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hnrz6" podStartSLOduration=3.166787298 podStartE2EDuration="4.211392088s" totalImagesPullingTime="962.72418ms" totalInitContainerRuntime="81.88061ms" isStatefulPod=true podCreationTimestamp="2026-08-04 17:53:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:26.210912749 +0000 UTC m=+61.901603825" watchObservedRunningTime="2026-08-04 17:53:26.211392088 +0000 UTC m=+61.902083138" Aug 04 17:53:26.524930 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:26.524850 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:26.658145 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:26.658112 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:26.967279 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:26.967236 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:26.986563 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:26.986532 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-78w5w"] Aug 04 17:53:26.989003 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:26.988980 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-78w5w" Aug 04 17:53:26.991658 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:26.991620 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Aug 04 17:53:26.991658 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:26.991639 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5wklc\"" Aug 04 17:53:27.004365 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:27.004345 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-78w5w"] Aug 04 17:53:27.038276 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:27.038212 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:27.095833 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:27.095800 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w57dw\" (UniqueName: \"kubernetes.io/projected/6a5eec29-205a-4c1a-8a0e-c156b7c6c334-kube-api-access-w57dw\") pod \"monitoring-plugin-6dffbbd67-78w5w\" (UID: \"6a5eec29-205a-4c1a-8a0e-c156b7c6c334\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-78w5w" Aug 04 17:53:27.096003 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:27.095907 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a5eec29-205a-4c1a-8a0e-c156b7c6c334-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-78w5w\" (UID: \"6a5eec29-205a-4c1a-8a0e-c156b7c6c334\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-78w5w" Aug 04 17:53:27.103119 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:27.103091 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:27.114977 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:27.114946 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bj5tc" Aug 04 17:53:27.197585 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:27.197084 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w57dw\" (UniqueName: \"kubernetes.io/projected/6a5eec29-205a-4c1a-8a0e-c156b7c6c334-kube-api-access-w57dw\") pod \"monitoring-plugin-6dffbbd67-78w5w\" (UID: \"6a5eec29-205a-4c1a-8a0e-c156b7c6c334\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-78w5w" Aug 04 17:53:27.197585 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:27.197189 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a5eec29-205a-4c1a-8a0e-c156b7c6c334-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-78w5w\" (UID: \"6a5eec29-205a-4c1a-8a0e-c156b7c6c334\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-78w5w" Aug 04 17:53:27.200326 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:27.200278 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6a5eec29-205a-4c1a-8a0e-c156b7c6c334-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-78w5w\" (UID: \"6a5eec29-205a-4c1a-8a0e-c156b7c6c334\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-78w5w" Aug 04 17:53:27.217847 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:27.217733 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w57dw\" (UniqueName: \"kubernetes.io/projected/6a5eec29-205a-4c1a-8a0e-c156b7c6c334-kube-api-access-w57dw\") pod \"monitoring-plugin-6dffbbd67-78w5w\" (UID: \"6a5eec29-205a-4c1a-8a0e-c156b7c6c334\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-78w5w" Aug 04 17:53:27.302069 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:27.301842 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-78w5w" Aug 04 17:53:27.461245 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:27.461197 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-78w5w"] Aug 04 17:53:27.907550 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:27.907509 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:27.976004 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:27.975917 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:28.014564 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:53:28.014529 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5eec29_205a_4c1a_8a0e_c156b7c6c334.slice/crio-96f6d6e0235465d1616b663fe8f0a31cbbcbc76d838f6ac1d703fc0abb62175c WatchSource:0}: Error finding container 96f6d6e0235465d1616b663fe8f0a31cbbcbc76d838f6ac1d703fc0abb62175c: Status 404 returned error can't find the container with id 96f6d6e0235465d1616b663fe8f0a31cbbcbc76d838f6ac1d703fc0abb62175c Aug 04 17:53:28.173106 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.173024 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-78w5w" event={"ID":"6a5eec29-205a-4c1a-8a0e-c156b7c6c334","Type":"ContainerStarted","Data":"96f6d6e0235465d1616b663fe8f0a31cbbcbc76d838f6ac1d703fc0abb62175c"} Aug 04 17:53:28.588175 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.588140 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:28.596958 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.594848 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 04 17:53:28.602843 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.601876 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.606481 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.605477 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Aug 04 17:53:28.608286 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.608264 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Aug 04 17:53:28.608604 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.608381 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Aug 04 17:53:28.608717 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.608701 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Aug 04 17:53:28.611702 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.608898 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Aug 04 17:53:28.611702 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.609139 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Aug 04 17:53:28.611702 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.609263 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Aug 04 17:53:28.611702 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.609393 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Aug 04 17:53:28.611702 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.609476 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Aug 04 17:53:28.611702 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.609613 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Aug 04 17:53:28.611702 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.609875 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-92kc8\"" Aug 04 17:53:28.611702 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.610113 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Aug 04 17:53:28.611702 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.610320 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Aug 04 17:53:28.611702 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.610616 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-40k3ra6v8s84b\"" Aug 04 17:53:28.611702 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.610870 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Aug 04 17:53:28.614554 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.614525 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Aug 04 17:53:28.620550 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.620523 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 04 17:53:28.626851 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.626827 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Aug 04 17:53:28.715080 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.714539 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.715080 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.714592 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.715080 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.714625 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.715080 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.714654 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvlpm\" (UniqueName: \"kubernetes.io/projected/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-kube-api-access-zvlpm\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.715080 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.714762 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.715080 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.714800 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.715080 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.714835 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.715080 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.714861 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-config-out\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.715080 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.714885 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.715080 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.714924 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.715080 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.714944 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-web-config\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.715080 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.714966 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.715080 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.715005 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.715080 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.715044 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-config\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.715080 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.715079 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.715883 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.715101 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.715883 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.715126 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.715883 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.715155 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.715883 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.715180 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.715883 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.715205 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.815879 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.815841 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.816036 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.815889 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.816036 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.815921 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.816036 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.815951 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.816036 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.815977 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.816036 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.816004 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.816282 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.816040 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.816282 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.816078 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.816282 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.816107 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.816282 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.816128 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvlpm\" (UniqueName: \"kubernetes.io/projected/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-kube-api-access-zvlpm\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.816282 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.816183 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.816282 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.816209 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.816282 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.816240 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.816282 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.816263 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-config-out\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.816636 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.816284 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.816636 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.816319 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.816636 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.816341 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-web-config\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.816636 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.816362 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.816636 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.816393 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.816636 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.816436 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-config\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.820225 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.817575 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.820225 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.818421 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.820225 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.819295 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.820225 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.819542 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-config\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.820225 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.819872 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.820547 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.820502 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.822067 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.821817 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.822305 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.822237 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.823035 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.823007 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.824533 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.824508 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.825931 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.825586 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.825931 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.825891 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-config-out\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.826076 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.826051 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.826602 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.826376 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.826602 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.826437 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.826602 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.826472 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.826856 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.826835 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.827945 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.827924 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.828522 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.828502 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-web-config\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.832213 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.832190 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvlpm\" (UniqueName: \"kubernetes.io/projected/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-kube-api-access-zvlpm\") pod \"prometheus-k8s-0\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:28.950528 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:28.950377 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:29.180436 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:29.180402 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" event={"ID":"d2e3e902-9da0-498d-87ff-3e9040b86081","Type":"ContainerStarted","Data":"791b40820777657eba3e1d4573a147bdbc1050b03fe2f88c47ae18ce4d259faa"} Aug 04 17:53:29.180611 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:29.180473 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" event={"ID":"d2e3e902-9da0-498d-87ff-3e9040b86081","Type":"ContainerStarted","Data":"62bc93c8e693d0a9e5c1df9f4d3e5783aa3ecadbe934dbbbeafa0543e59e5605"} Aug 04 17:53:29.180611 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:29.180514 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" event={"ID":"d2e3e902-9da0-498d-87ff-3e9040b86081","Type":"ContainerStarted","Data":"6eb905e50d30dbb9008da8ade5bbab33ae40ac2e808fde1194d94c9e543ffe74"} Aug 04 17:53:29.183140 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:29.183114 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b","Type":"ContainerStarted","Data":"f9fab62bcf935b3e3aaf6615c1fa343a28925277715170fdb390cf933c69c0fb"} Aug 04 17:53:29.183468 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:29.183449 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b","Type":"ContainerStarted","Data":"b49e11430b35b61ea25aac446425429ccbc06e87415827a5f78235ca75ab37e0"} Aug 04 17:53:29.183540 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:29.183502 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b","Type":"ContainerStarted","Data":"1fb39df25c74c331256afefdeda13f11ea155f2eaaa4ee941a9658d25cbf790f"} Aug 04 17:53:29.943056 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:29.943017 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:30.043708 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:30.043677 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:30.634198 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:30.634162 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/096b3bbb-ead1-451f-9f3a-e5a4add161ad-original-pull-secret\") pod \"global-pull-secret-syncer-xbhvg\" (UID: \"096b3bbb-ead1-451f-9f3a-e5a4add161ad\") " pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:53:30.634372 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:30.634219 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aa144f2-e9b4-4162-be3f-6cd9405ac466-metrics-certs\") pod \"network-metrics-daemon-mpgkn\" (UID: \"8aa144f2-e9b4-4162-be3f-6cd9405ac466\") " pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:53:30.636896 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:30.636871 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 17:53:30.636999 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:30.636905 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 17:53:30.647565 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:30.647540 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/096b3bbb-ead1-451f-9f3a-e5a4add161ad-original-pull-secret\") pod \"global-pull-secret-syncer-xbhvg\" (UID: \"096b3bbb-ead1-451f-9f3a-e5a4add161ad\") " pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:53:30.647654 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:30.647583 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aa144f2-e9b4-4162-be3f-6cd9405ac466-metrics-certs\") pod \"network-metrics-daemon-mpgkn\" (UID: \"8aa144f2-e9b4-4162-be3f-6cd9405ac466\") " pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:53:30.728866 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:30.728843 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-smv6k\"" Aug 04 17:53:30.734943 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:30.734923 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-479kg\" (UniqueName: \"kubernetes.io/projected/4bb95b54-5442-43b6-befa-8ecf7b8f4781-kube-api-access-479kg\") pod \"network-check-target-9z2cq\" (UID: \"4bb95b54-5442-43b6-befa-8ecf7b8f4781\") " pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:53:30.736824 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:30.736803 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpgkn" Aug 04 17:53:30.737608 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:30.737591 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 17:53:30.747920 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:30.747900 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 17:53:30.758980 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:30.758963 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-479kg\" (UniqueName: \"kubernetes.io/projected/4bb95b54-5442-43b6-befa-8ecf7b8f4781-kube-api-access-479kg\") pod \"network-check-target-9z2cq\" (UID: \"4bb95b54-5442-43b6-befa-8ecf7b8f4781\") " pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:53:30.828767 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:30.828740 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xbhvg" Aug 04 17:53:30.838295 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:30.838270 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-f7jt5\"" Aug 04 17:53:30.845895 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:30.845869 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:53:31.348190 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:31.348162 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:31.424292 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:31.424262 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:32.115415 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:32.115385 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:32.220141 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:32.220106 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:33.097480 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:33.097434 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:33.196525 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:33.196466 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:34.998680 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:34.998626 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:35.114145 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:35.114114 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:36.170207 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:36.170134 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:36.235480 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:36.235444 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:36.539838 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:36.538981 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xbhvg"] Aug 04 17:53:36.550534 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:53:36.550495 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096b3bbb_ead1_451f_9f3a_e5a4add161ad.slice/crio-031fbd762c8cc1148910ff2d690299af61cdea6503bf603accac9b7e637bac78 WatchSource:0}: Error finding container 031fbd762c8cc1148910ff2d690299af61cdea6503bf603accac9b7e637bac78: Status 404 returned error can't find the container with id 031fbd762c8cc1148910ff2d690299af61cdea6503bf603accac9b7e637bac78 Aug 04 17:53:36.591524 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:36.591125 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9z2cq"] Aug 04 17:53:36.597440 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:53:36.597395 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb95b54_5442_43b6_befa_8ecf7b8f4781.slice/crio-e4f85731bf55710310029e702ca059541331e2356d7387d70816f23c4c1b3f1f WatchSource:0}: Error finding container e4f85731bf55710310029e702ca059541331e2356d7387d70816f23c4c1b3f1f: Status 404 returned error can't find the container with id e4f85731bf55710310029e702ca059541331e2356d7387d70816f23c4c1b3f1f Aug 04 17:53:36.760401 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:36.760321 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mpgkn"] Aug 04 17:53:36.763639 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:53:36.763614 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa144f2_e9b4_4162_be3f_6cd9405ac466.slice/crio-2055c5923df937a40a93405c68e76f2283c3e291a94d95bd65d902e1fed6d360 WatchSource:0}: Error finding container 2055c5923df937a40a93405c68e76f2283c3e291a94d95bd65d902e1fed6d360: Status 404 returned error can't find the container with id 2055c5923df937a40a93405c68e76f2283c3e291a94d95bd65d902e1fed6d360 Aug 04 17:53:36.780002 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:36.779975 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 04 17:53:36.783117 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:53:36.783086 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0d35e9_7cc7_45c0_ba83_b581c25e172e.slice/crio-1ff13c4be5374ddd59786f1cec4ae81eac5a4b90a8a083d9401de89b5db6077b WatchSource:0}: Error finding container 1ff13c4be5374ddd59786f1cec4ae81eac5a4b90a8a083d9401de89b5db6077b: Status 404 returned error can't find the container with id 1ff13c4be5374ddd59786f1cec4ae81eac5a4b90a8a083d9401de89b5db6077b Aug 04 17:53:37.094735 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:37.094641 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:37.231491 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:37.231026 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-gf5hr" event={"ID":"48ff3a32-3414-4512-a8d8-7918e3ae955a","Type":"ContainerStarted","Data":"5e1956b06cfb9e3c4b6c9e904dca4d3acccf7a4b81faad85fc90675da050a94b"} Aug 04 17:53:37.232118 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:37.232092 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-77c974ffcf-gf5hr" Aug 04 17:53:37.235720 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:37.235499 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-78w5w" event={"ID":"6a5eec29-205a-4c1a-8a0e-c156b7c6c334","Type":"ContainerStarted","Data":"3ee4a28d43a5cbd99c9527b561ebaba5982897982312ead60ca770e75b893108"} Aug 04 17:53:37.236205 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:37.236149 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-78w5w" Aug 04 17:53:37.242737 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:37.242715 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-78w5w" Aug 04 17:53:37.242917 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:37.242893 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9z2cq" event={"ID":"4bb95b54-5442-43b6-befa-8ecf7b8f4781","Type":"ContainerStarted","Data":"e4f85731bf55710310029e702ca059541331e2356d7387d70816f23c4c1b3f1f"} Aug 04 17:53:37.247592 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:37.246622 2572 generic.go:356] "Generic (PLEG): container finished" podID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerID="5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4" exitCode=0 Aug 04 17:53:37.247592 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:37.246711 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a0d35e9-7cc7-45c0-ba83-b581c25e172e","Type":"ContainerDied","Data":"5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4"} Aug 04 17:53:37.247592 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:37.246769 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a0d35e9-7cc7-45c0-ba83-b581c25e172e","Type":"ContainerStarted","Data":"1ff13c4be5374ddd59786f1cec4ae81eac5a4b90a8a083d9401de89b5db6077b"} Aug 04 17:53:37.247821 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:37.247772 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-77c974ffcf-gf5hr" Aug 04 17:53:37.250207 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:37.250041 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mpgkn" event={"ID":"8aa144f2-e9b4-4162-be3f-6cd9405ac466","Type":"ContainerStarted","Data":"2055c5923df937a40a93405c68e76f2283c3e291a94d95bd65d902e1fed6d360"} Aug 04 17:53:37.256965 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:37.256085 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-77c974ffcf-gf5hr" podStartSLOduration=2.246832308 podStartE2EDuration="20.256068658s" totalImagesPullingTime="18.00923635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:53:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:37.255049526 +0000 UTC m=+72.945740592" watchObservedRunningTime="2026-08-04 17:53:37.256068658 +0000 UTC m=+72.946759710" Aug 04 17:53:37.256965 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:37.256498 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" event={"ID":"d2e3e902-9da0-498d-87ff-3e9040b86081","Type":"ContainerStarted","Data":"9be7bab25f11afa8cc03f211a2e5bf335a2fff95b429b0c3db62148e4cda84de"} Aug 04 17:53:37.256965 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:37.256593 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" event={"ID":"d2e3e902-9da0-498d-87ff-3e9040b86081","Type":"ContainerStarted","Data":"2aff96aa189f0ed3dbcf966769acce99e314767501ffd697664a8507d848726b"} Aug 04 17:53:37.256965 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:37.256655 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" event={"ID":"d2e3e902-9da0-498d-87ff-3e9040b86081","Type":"ContainerStarted","Data":"b6837ba79540e25750546575a64f11f3c8646886babfbe4b873c930ab7c550c3"} Aug 04 17:53:37.261168 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:37.261131 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xbhvg" event={"ID":"096b3bbb-ead1-451f-9f3a-e5a4add161ad","Type":"ContainerStarted","Data":"031fbd762c8cc1148910ff2d690299af61cdea6503bf603accac9b7e637bac78"} Aug 04 17:53:37.267294 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:37.267268 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b","Type":"ContainerStarted","Data":"7c528b6a7359568b574d79d15125b2b7844e3ac889acb3358cf6c515dd6cb10b"} Aug 04 17:53:37.267396 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:37.267339 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b","Type":"ContainerStarted","Data":"aeaed168ce65fffe0e3fa174562477a8bf49061ae442a560c160d751562963e3"} Aug 04 17:53:37.267396 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:37.267385 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b","Type":"ContainerStarted","Data":"50837385af84d677faf7c7bc86fab5a249c94e1e1c4daeceaed14b72b47ff458"} Aug 04 17:53:37.274953 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:37.274903 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-78w5w" podStartSLOduration=2.93171468 podStartE2EDuration="11.274884358s" totalImagesPullingTime="8.343169678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:53:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:37.27300091 +0000 UTC m=+72.963691984" watchObservedRunningTime="2026-08-04 17:53:37.274884358 +0000 UTC m=+72.965575400" Aug 04 17:53:37.371933 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:37.371827 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" podStartSLOduration=2.113458142 podStartE2EDuration="12.371808929s" totalImagesPullingTime="10.258350787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:53:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:37.370460381 +0000 UTC m=+73.061151572" watchObservedRunningTime="2026-08-04 17:53:37.371808929 +0000 UTC m=+73.062499979" Aug 04 17:53:37.403479 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:37.403418 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=10.913524685 podStartE2EDuration="14.403398566s" totalImagesPullingTime="3.471433595s" totalInitContainerRuntime="18.440286ms" isStatefulPod=false podCreationTimestamp="2026-08-04 17:53:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:37.401736984 +0000 UTC m=+73.092428047" watchObservedRunningTime="2026-08-04 17:53:37.403398566 +0000 UTC m=+73.094089619" Aug 04 17:53:38.276982 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:38.276884 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:38.282895 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:38.282872 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5c8d9fcf-f6b4w" Aug 04 17:53:39.114803 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:39.114760 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:39.222233 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:39.222182 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:40.303288 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:40.303212 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mpgkn" event={"ID":"8aa144f2-e9b4-4162-be3f-6cd9405ac466","Type":"ContainerStarted","Data":"27579d8e59d7077c82c9bdc53263ed2d3912bd5b115c4a0dadd6378c83803270"} Aug 04 17:53:40.303288 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:40.303271 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mpgkn" event={"ID":"8aa144f2-e9b4-4162-be3f-6cd9405ac466","Type":"ContainerStarted","Data":"2631d1ab19637510ab496058d85672cd4f8b34c82e10beb5b5b23d44402fa1bb"} Aug 04 17:53:40.344163 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:40.342594 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mpgkn" podStartSLOduration=72.879275708 podStartE2EDuration="1m15.342572544s" totalImagesPullingTime="2.463296836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:40.342228848 +0000 UTC m=+76.032919900" watchObservedRunningTime="2026-08-04 17:53:40.342572544 +0000 UTC m=+76.033263596" Aug 04 17:53:40.489400 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:40.489260 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:40.589162 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:40.589018 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:40.698082 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:40.697968 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:41.178940 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:41.178839 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:41.326491 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:41.326414 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:41.460874 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:41.460779 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:43.137469 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:43.137430 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:43.244273 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:43.244197 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:44.188153 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:44.188105 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:44.265632 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:44.265593 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:46.341511 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:46.341480 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9z2cq" event={"ID":"4bb95b54-5442-43b6-befa-8ecf7b8f4781","Type":"ContainerStarted","Data":"5daf2865df583521db1946c4d7c2a9f766fa5455f55a000c2b80e745a2d2e1fe"} Aug 04 17:53:46.344165 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:46.344143 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a0d35e9-7cc7-45c0-ba83-b581c25e172e","Type":"ContainerStarted","Data":"a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7"} Aug 04 17:53:46.344283 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:46.344205 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a0d35e9-7cc7-45c0-ba83-b581c25e172e","Type":"ContainerStarted","Data":"40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea"} Aug 04 17:53:46.344283 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:46.344248 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a0d35e9-7cc7-45c0-ba83-b581c25e172e","Type":"ContainerStarted","Data":"ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7"} Aug 04 17:53:46.344388 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:46.344284 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a0d35e9-7cc7-45c0-ba83-b581c25e172e","Type":"ContainerStarted","Data":"e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173"} Aug 04 17:53:46.345488 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:46.345466 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xbhvg" event={"ID":"096b3bbb-ead1-451f-9f3a-e5a4add161ad","Type":"ContainerStarted","Data":"1493eb40eb6bc7246c85c8d4978a708101928303c7421f783494580360bcd60c"} Aug 04 17:53:46.347186 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:46.347172 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:53:46.395366 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:46.395324 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9z2cq" podStartSLOduration=72.181080403 podStartE2EDuration="1m21.395309934s" totalImagesPullingTime="9.214229531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:46.36934236 +0000 UTC m=+82.060033411" watchObservedRunningTime="2026-08-04 17:53:46.395309934 +0000 UTC m=+82.086000985" Aug 04 17:53:46.395930 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:46.395905 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xbhvg" podStartSLOduration=72.123530721 podStartE2EDuration="1m21.395895232s" totalImagesPullingTime="9.272364511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:46.394979047 +0000 UTC m=+82.085670098" watchObservedRunningTime="2026-08-04 17:53:46.395895232 +0000 UTC m=+82.086586283" Aug 04 17:53:47.352136 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:47.352098 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a0d35e9-7cc7-45c0-ba83-b581c25e172e","Type":"ContainerStarted","Data":"e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19"} Aug 04 17:53:47.352622 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:47.352171 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a0d35e9-7cc7-45c0-ba83-b581c25e172e","Type":"ContainerStarted","Data":"d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7"} Aug 04 17:53:47.406158 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:47.406113 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=10.813937548 podStartE2EDuration="19.406097764s" totalImagesPullingTime="8.564902906s" totalInitContainerRuntime="27.25731ms" isStatefulPod=false podCreationTimestamp="2026-08-04 17:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:53:47.402509651 +0000 UTC m=+83.093200701" watchObservedRunningTime="2026-08-04 17:53:47.406097764 +0000 UTC m=+83.096788814" Aug 04 17:53:48.951080 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:48.951046 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:53:56.144343 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:56.144301 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:53:56.203390 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:53:56.203354 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:54:16.118195 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:16.118139 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:54:17.354626 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:17.354598 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9z2cq" Aug 04 17:54:28.950978 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:28.950947 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:29.008828 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:29.008799 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:29.528864 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:29.528835 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:37.993430 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:37.993389 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:54:41.627389 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:41.627354 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:54:41.825323 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:41.825290 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:54:42.712916 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:42.712881 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 17:54:42.713295 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:42.713163 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="alertmanager" containerID="cri-o://1fb39df25c74c331256afefdeda13f11ea155f2eaaa4ee941a9658d25cbf790f" gracePeriod=120 Aug 04 17:54:42.713295 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:42.713235 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="kube-rbac-proxy-metric" containerID="cri-o://aeaed168ce65fffe0e3fa174562477a8bf49061ae442a560c160d751562963e3" gracePeriod=120 Aug 04 17:54:42.713295 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:42.713255 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="prom-label-proxy" containerID="cri-o://7c528b6a7359568b574d79d15125b2b7844e3ac889acb3358cf6c515dd6cb10b" gracePeriod=120 Aug 04 17:54:42.713510 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:42.713271 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="kube-rbac-proxy-web" containerID="cri-o://f9fab62bcf935b3e3aaf6615c1fa343a28925277715170fdb390cf933c69c0fb" gracePeriod=120 Aug 04 17:54:42.713510 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:42.713312 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="kube-rbac-proxy" containerID="cri-o://50837385af84d677faf7c7bc86fab5a249c94e1e1c4daeceaed14b72b47ff458" gracePeriod=120 Aug 04 17:54:42.713510 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:42.713343 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="config-reloader" containerID="cri-o://b49e11430b35b61ea25aac446425429ccbc06e87415827a5f78235ca75ab37e0" gracePeriod=120 Aug 04 17:54:43.568507 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:43.568473 2572 generic.go:356] "Generic (PLEG): container finished" podID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerID="7c528b6a7359568b574d79d15125b2b7844e3ac889acb3358cf6c515dd6cb10b" exitCode=0 Aug 04 17:54:43.568507 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:43.568501 2572 generic.go:356] "Generic (PLEG): container finished" podID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerID="50837385af84d677faf7c7bc86fab5a249c94e1e1c4daeceaed14b72b47ff458" exitCode=0 Aug 04 17:54:43.568507 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:43.568507 2572 generic.go:356] "Generic (PLEG): container finished" podID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerID="b49e11430b35b61ea25aac446425429ccbc06e87415827a5f78235ca75ab37e0" exitCode=0 Aug 04 17:54:43.568507 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:43.568512 2572 generic.go:356] "Generic (PLEG): container finished" podID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerID="1fb39df25c74c331256afefdeda13f11ea155f2eaaa4ee941a9658d25cbf790f" exitCode=0 Aug 04 17:54:43.568796 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:43.568545 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b","Type":"ContainerDied","Data":"7c528b6a7359568b574d79d15125b2b7844e3ac889acb3358cf6c515dd6cb10b"} Aug 04 17:54:43.568796 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:43.568652 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b","Type":"ContainerDied","Data":"50837385af84d677faf7c7bc86fab5a249c94e1e1c4daeceaed14b72b47ff458"} Aug 04 17:54:43.568796 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:43.568697 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b","Type":"ContainerDied","Data":"b49e11430b35b61ea25aac446425429ccbc06e87415827a5f78235ca75ab37e0"} Aug 04 17:54:43.568796 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:43.568723 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b","Type":"ContainerDied","Data":"1fb39df25c74c331256afefdeda13f11ea155f2eaaa4ee941a9658d25cbf790f"} Aug 04 17:54:43.972945 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:43.972922 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.070549 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.070519 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-alertmanager-main-db\") pod \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " Aug 04 17:54:44.070754 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.070564 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-main-tls\") pod \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " Aug 04 17:54:44.070754 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.070593 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-alertmanager-trusted-ca-bundle\") pod \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " Aug 04 17:54:44.070754 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.070625 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-kube-rbac-proxy-web\") pod \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " Aug 04 17:54:44.070754 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.070651 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-cluster-tls-config\") pod \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " Aug 04 17:54:44.070754 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.070695 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-kube-rbac-proxy\") pod \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " Aug 04 17:54:44.070754 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.070726 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-metrics-client-ca\") pod \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " Aug 04 17:54:44.071107 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.070775 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-web-config\") pod \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " Aug 04 17:54:44.071107 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.070809 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-config-out\") pod \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " Aug 04 17:54:44.071107 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.070843 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-config-volume\") pod \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " Aug 04 17:54:44.071107 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.070868 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6shv\" (UniqueName: \"kubernetes.io/projected/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-kube-api-access-n6shv\") pod \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " Aug 04 17:54:44.071107 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.070917 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " Aug 04 17:54:44.071107 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.070932 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" (UID: "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 17:54:44.071107 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.070958 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-tls-assets\") pod \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\" (UID: \"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b\") " Aug 04 17:54:44.071107 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.071079 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" (UID: "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:54:44.071500 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.071350 2572 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-alertmanager-main-db\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:44.071500 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.071311 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" (UID: "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:54:44.071500 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.071373 2572 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:44.074318 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.074274 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-kube-api-access-n6shv" (OuterVolumeSpecName: "kube-api-access-n6shv") pod "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" (UID: "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b"). InnerVolumeSpecName "kube-api-access-n6shv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:54:44.074948 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.074901 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" (UID: "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:54:44.075240 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.075212 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-config-out" (OuterVolumeSpecName: "config-out") pod "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" (UID: "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 17:54:44.075240 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.075211 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" (UID: "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:54:44.075424 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.075339 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" (UID: "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:54:44.075479 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.075455 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" (UID: "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:54:44.076028 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.076000 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" (UID: "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:54:44.076210 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.076194 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-config-volume" (OuterVolumeSpecName: "config-volume") pod "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" (UID: "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:54:44.080119 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.080097 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" (UID: "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:54:44.086686 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.086647 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-web-config" (OuterVolumeSpecName: "web-config") pod "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" (UID: "15a69893-7efd-4ce6-98da-d6bbe0a2ec3b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:54:44.172197 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.172164 2572 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-tls-assets\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:44.172197 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.172190 2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-main-tls\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:44.172197 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.172201 2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:44.172388 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.172212 2572 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-cluster-tls-config\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:44.172388 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.172220 2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:44.172388 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.172229 2572 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-metrics-client-ca\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:44.172388 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.172237 2572 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-web-config\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:44.172388 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.172244 2572 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-config-out\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:44.172388 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.172251 2572 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-config-volume\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:44.172388 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.172260 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6shv\" (UniqueName: \"kubernetes.io/projected/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-kube-api-access-n6shv\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:44.172388 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.172268 2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:44.575508 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.575477 2572 generic.go:356] "Generic (PLEG): container finished" podID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerID="aeaed168ce65fffe0e3fa174562477a8bf49061ae442a560c160d751562963e3" exitCode=0 Aug 04 17:54:44.575508 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.575501 2572 generic.go:356] "Generic (PLEG): container finished" podID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerID="f9fab62bcf935b3e3aaf6615c1fa343a28925277715170fdb390cf933c69c0fb" exitCode=0 Aug 04 17:54:44.575687 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.575523 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b","Type":"ContainerDied","Data":"aeaed168ce65fffe0e3fa174562477a8bf49061ae442a560c160d751562963e3"} Aug 04 17:54:44.575687 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.575552 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.575687 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.575608 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b","Type":"ContainerDied","Data":"f9fab62bcf935b3e3aaf6615c1fa343a28925277715170fdb390cf933c69c0fb"} Aug 04 17:54:44.575687 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.575641 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"15a69893-7efd-4ce6-98da-d6bbe0a2ec3b","Type":"ContainerDied","Data":"7da4d850e43744928fb4a64fcdd640088a9d884ec2a4c231064e226c572660f8"} Aug 04 17:54:44.575835 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.575692 2572 scope.go:117] "RemoveContainer" containerID="7c528b6a7359568b574d79d15125b2b7844e3ac889acb3358cf6c515dd6cb10b" Aug 04 17:54:44.584365 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.584291 2572 scope.go:117] "RemoveContainer" containerID="aeaed168ce65fffe0e3fa174562477a8bf49061ae442a560c160d751562963e3" Aug 04 17:54:44.591291 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.591276 2572 scope.go:117] "RemoveContainer" containerID="50837385af84d677faf7c7bc86fab5a249c94e1e1c4daeceaed14b72b47ff458" Aug 04 17:54:44.597864 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.597849 2572 scope.go:117] "RemoveContainer" containerID="f9fab62bcf935b3e3aaf6615c1fa343a28925277715170fdb390cf933c69c0fb" Aug 04 17:54:44.604619 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.604594 2572 scope.go:117] "RemoveContainer" containerID="b49e11430b35b61ea25aac446425429ccbc06e87415827a5f78235ca75ab37e0" Aug 04 17:54:44.607020 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.606978 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 17:54:44.609997 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.609978 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 17:54:44.613001 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.612986 2572 scope.go:117] "RemoveContainer" containerID="1fb39df25c74c331256afefdeda13f11ea155f2eaaa4ee941a9658d25cbf790f" Aug 04 17:54:44.619769 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.619752 2572 scope.go:117] "RemoveContainer" containerID="30247426d51c290e26690504fa14b92ffd999777a3a6f515efe362698500260c" Aug 04 17:54:44.626386 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.626365 2572 scope.go:117] "RemoveContainer" containerID="7c528b6a7359568b574d79d15125b2b7844e3ac889acb3358cf6c515dd6cb10b" Aug 04 17:54:44.626647 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:54:44.626626 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c528b6a7359568b574d79d15125b2b7844e3ac889acb3358cf6c515dd6cb10b\": container with ID starting with 7c528b6a7359568b574d79d15125b2b7844e3ac889acb3358cf6c515dd6cb10b not found: ID does not exist" containerID="7c528b6a7359568b574d79d15125b2b7844e3ac889acb3358cf6c515dd6cb10b" Aug 04 17:54:44.626763 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.626654 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c528b6a7359568b574d79d15125b2b7844e3ac889acb3358cf6c515dd6cb10b"} err="failed to get container status \"7c528b6a7359568b574d79d15125b2b7844e3ac889acb3358cf6c515dd6cb10b\": rpc error: code = NotFound desc = could not find container \"7c528b6a7359568b574d79d15125b2b7844e3ac889acb3358cf6c515dd6cb10b\": container with ID starting with 7c528b6a7359568b574d79d15125b2b7844e3ac889acb3358cf6c515dd6cb10b not found: ID does not exist" Aug 04 17:54:44.626763 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.626715 2572 scope.go:117] "RemoveContainer" containerID="aeaed168ce65fffe0e3fa174562477a8bf49061ae442a560c160d751562963e3" Aug 04 17:54:44.626931 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:54:44.626911 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeaed168ce65fffe0e3fa174562477a8bf49061ae442a560c160d751562963e3\": container with ID starting with aeaed168ce65fffe0e3fa174562477a8bf49061ae442a560c160d751562963e3 not found: ID does not exist" containerID="aeaed168ce65fffe0e3fa174562477a8bf49061ae442a560c160d751562963e3" Aug 04 17:54:44.626995 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.626940 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeaed168ce65fffe0e3fa174562477a8bf49061ae442a560c160d751562963e3"} err="failed to get container status \"aeaed168ce65fffe0e3fa174562477a8bf49061ae442a560c160d751562963e3\": rpc error: code = NotFound desc = could not find container \"aeaed168ce65fffe0e3fa174562477a8bf49061ae442a560c160d751562963e3\": container with ID starting with aeaed168ce65fffe0e3fa174562477a8bf49061ae442a560c160d751562963e3 not found: ID does not exist" Aug 04 17:54:44.626995 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.626964 2572 scope.go:117] "RemoveContainer" containerID="50837385af84d677faf7c7bc86fab5a249c94e1e1c4daeceaed14b72b47ff458" Aug 04 17:54:44.627185 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:54:44.627168 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50837385af84d677faf7c7bc86fab5a249c94e1e1c4daeceaed14b72b47ff458\": container with ID starting with 50837385af84d677faf7c7bc86fab5a249c94e1e1c4daeceaed14b72b47ff458 not found: ID does not exist" containerID="50837385af84d677faf7c7bc86fab5a249c94e1e1c4daeceaed14b72b47ff458" Aug 04 17:54:44.627219 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.627189 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50837385af84d677faf7c7bc86fab5a249c94e1e1c4daeceaed14b72b47ff458"} err="failed to get container status \"50837385af84d677faf7c7bc86fab5a249c94e1e1c4daeceaed14b72b47ff458\": rpc error: code = NotFound desc = could not find container \"50837385af84d677faf7c7bc86fab5a249c94e1e1c4daeceaed14b72b47ff458\": container with ID starting with 50837385af84d677faf7c7bc86fab5a249c94e1e1c4daeceaed14b72b47ff458 not found: ID does not exist" Aug 04 17:54:44.627219 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.627201 2572 scope.go:117] "RemoveContainer" containerID="f9fab62bcf935b3e3aaf6615c1fa343a28925277715170fdb390cf933c69c0fb" Aug 04 17:54:44.627392 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:54:44.627378 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9fab62bcf935b3e3aaf6615c1fa343a28925277715170fdb390cf933c69c0fb\": container with ID starting with f9fab62bcf935b3e3aaf6615c1fa343a28925277715170fdb390cf933c69c0fb not found: ID does not exist" containerID="f9fab62bcf935b3e3aaf6615c1fa343a28925277715170fdb390cf933c69c0fb" Aug 04 17:54:44.627434 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.627395 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9fab62bcf935b3e3aaf6615c1fa343a28925277715170fdb390cf933c69c0fb"} err="failed to get container status \"f9fab62bcf935b3e3aaf6615c1fa343a28925277715170fdb390cf933c69c0fb\": rpc error: code = NotFound desc = could not find container \"f9fab62bcf935b3e3aaf6615c1fa343a28925277715170fdb390cf933c69c0fb\": container with ID starting with f9fab62bcf935b3e3aaf6615c1fa343a28925277715170fdb390cf933c69c0fb not found: ID does not exist" Aug 04 17:54:44.627434 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.627407 2572 scope.go:117] "RemoveContainer" containerID="b49e11430b35b61ea25aac446425429ccbc06e87415827a5f78235ca75ab37e0" Aug 04 17:54:44.627617 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:54:44.627596 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b49e11430b35b61ea25aac446425429ccbc06e87415827a5f78235ca75ab37e0\": container with ID starting with b49e11430b35b61ea25aac446425429ccbc06e87415827a5f78235ca75ab37e0 not found: ID does not exist" containerID="b49e11430b35b61ea25aac446425429ccbc06e87415827a5f78235ca75ab37e0" Aug 04 17:54:44.627656 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.627626 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b49e11430b35b61ea25aac446425429ccbc06e87415827a5f78235ca75ab37e0"} err="failed to get container status \"b49e11430b35b61ea25aac446425429ccbc06e87415827a5f78235ca75ab37e0\": rpc error: code = NotFound desc = could not find container \"b49e11430b35b61ea25aac446425429ccbc06e87415827a5f78235ca75ab37e0\": container with ID starting with b49e11430b35b61ea25aac446425429ccbc06e87415827a5f78235ca75ab37e0 not found: ID does not exist" Aug 04 17:54:44.627656 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.627642 2572 scope.go:117] "RemoveContainer" containerID="1fb39df25c74c331256afefdeda13f11ea155f2eaaa4ee941a9658d25cbf790f" Aug 04 17:54:44.627867 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:54:44.627852 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fb39df25c74c331256afefdeda13f11ea155f2eaaa4ee941a9658d25cbf790f\": container with ID starting with 1fb39df25c74c331256afefdeda13f11ea155f2eaaa4ee941a9658d25cbf790f not found: ID does not exist" containerID="1fb39df25c74c331256afefdeda13f11ea155f2eaaa4ee941a9658d25cbf790f" Aug 04 17:54:44.627903 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.627872 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fb39df25c74c331256afefdeda13f11ea155f2eaaa4ee941a9658d25cbf790f"} err="failed to get container status \"1fb39df25c74c331256afefdeda13f11ea155f2eaaa4ee941a9658d25cbf790f\": rpc error: code = NotFound desc = could not find container \"1fb39df25c74c331256afefdeda13f11ea155f2eaaa4ee941a9658d25cbf790f\": container with ID starting with 1fb39df25c74c331256afefdeda13f11ea155f2eaaa4ee941a9658d25cbf790f not found: ID does not exist" Aug 04 17:54:44.627903 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.627883 2572 scope.go:117] "RemoveContainer" containerID="30247426d51c290e26690504fa14b92ffd999777a3a6f515efe362698500260c" Aug 04 17:54:44.628073 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:54:44.628059 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30247426d51c290e26690504fa14b92ffd999777a3a6f515efe362698500260c\": container with ID starting with 30247426d51c290e26690504fa14b92ffd999777a3a6f515efe362698500260c not found: ID does not exist" containerID="30247426d51c290e26690504fa14b92ffd999777a3a6f515efe362698500260c" Aug 04 17:54:44.628107 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.628076 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30247426d51c290e26690504fa14b92ffd999777a3a6f515efe362698500260c"} err="failed to get container status \"30247426d51c290e26690504fa14b92ffd999777a3a6f515efe362698500260c\": rpc error: code = NotFound desc = could not find container \"30247426d51c290e26690504fa14b92ffd999777a3a6f515efe362698500260c\": container with ID starting with 30247426d51c290e26690504fa14b92ffd999777a3a6f515efe362698500260c not found: ID does not exist" Aug 04 17:54:44.628107 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.628088 2572 scope.go:117] "RemoveContainer" containerID="7c528b6a7359568b574d79d15125b2b7844e3ac889acb3358cf6c515dd6cb10b" Aug 04 17:54:44.628272 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.628257 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c528b6a7359568b574d79d15125b2b7844e3ac889acb3358cf6c515dd6cb10b"} err="failed to get container status \"7c528b6a7359568b574d79d15125b2b7844e3ac889acb3358cf6c515dd6cb10b\": rpc error: code = NotFound desc = could not find container \"7c528b6a7359568b574d79d15125b2b7844e3ac889acb3358cf6c515dd6cb10b\": container with ID starting with 7c528b6a7359568b574d79d15125b2b7844e3ac889acb3358cf6c515dd6cb10b not found: ID does not exist" Aug 04 17:54:44.628347 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.628273 2572 scope.go:117] "RemoveContainer" containerID="aeaed168ce65fffe0e3fa174562477a8bf49061ae442a560c160d751562963e3" Aug 04 17:54:44.628448 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.628433 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeaed168ce65fffe0e3fa174562477a8bf49061ae442a560c160d751562963e3"} err="failed to get container status \"aeaed168ce65fffe0e3fa174562477a8bf49061ae442a560c160d751562963e3\": rpc error: code = NotFound desc = could not find container \"aeaed168ce65fffe0e3fa174562477a8bf49061ae442a560c160d751562963e3\": container with ID starting with aeaed168ce65fffe0e3fa174562477a8bf49061ae442a560c160d751562963e3 not found: ID does not exist" Aug 04 17:54:44.628486 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.628448 2572 scope.go:117] "RemoveContainer" containerID="50837385af84d677faf7c7bc86fab5a249c94e1e1c4daeceaed14b72b47ff458" Aug 04 17:54:44.628621 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.628606 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50837385af84d677faf7c7bc86fab5a249c94e1e1c4daeceaed14b72b47ff458"} err="failed to get container status \"50837385af84d677faf7c7bc86fab5a249c94e1e1c4daeceaed14b72b47ff458\": rpc error: code = NotFound desc = could not find container \"50837385af84d677faf7c7bc86fab5a249c94e1e1c4daeceaed14b72b47ff458\": container with ID starting with 50837385af84d677faf7c7bc86fab5a249c94e1e1c4daeceaed14b72b47ff458 not found: ID does not exist" Aug 04 17:54:44.628682 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.628622 2572 scope.go:117] "RemoveContainer" containerID="f9fab62bcf935b3e3aaf6615c1fa343a28925277715170fdb390cf933c69c0fb" Aug 04 17:54:44.628848 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.628833 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9fab62bcf935b3e3aaf6615c1fa343a28925277715170fdb390cf933c69c0fb"} err="failed to get container status \"f9fab62bcf935b3e3aaf6615c1fa343a28925277715170fdb390cf933c69c0fb\": rpc error: code = NotFound desc = could not find container \"f9fab62bcf935b3e3aaf6615c1fa343a28925277715170fdb390cf933c69c0fb\": container with ID starting with f9fab62bcf935b3e3aaf6615c1fa343a28925277715170fdb390cf933c69c0fb not found: ID does not exist" Aug 04 17:54:44.628888 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.628848 2572 scope.go:117] "RemoveContainer" containerID="b49e11430b35b61ea25aac446425429ccbc06e87415827a5f78235ca75ab37e0" Aug 04 17:54:44.629024 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.629009 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b49e11430b35b61ea25aac446425429ccbc06e87415827a5f78235ca75ab37e0"} err="failed to get container status \"b49e11430b35b61ea25aac446425429ccbc06e87415827a5f78235ca75ab37e0\": rpc error: code = NotFound desc = could not find container \"b49e11430b35b61ea25aac446425429ccbc06e87415827a5f78235ca75ab37e0\": container with ID starting with b49e11430b35b61ea25aac446425429ccbc06e87415827a5f78235ca75ab37e0 not found: ID does not exist" Aug 04 17:54:44.629062 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.629024 2572 scope.go:117] "RemoveContainer" containerID="1fb39df25c74c331256afefdeda13f11ea155f2eaaa4ee941a9658d25cbf790f" Aug 04 17:54:44.629198 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.629183 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fb39df25c74c331256afefdeda13f11ea155f2eaaa4ee941a9658d25cbf790f"} err="failed to get container status \"1fb39df25c74c331256afefdeda13f11ea155f2eaaa4ee941a9658d25cbf790f\": rpc error: code = NotFound desc = could not find container \"1fb39df25c74c331256afefdeda13f11ea155f2eaaa4ee941a9658d25cbf790f\": container with ID starting with 1fb39df25c74c331256afefdeda13f11ea155f2eaaa4ee941a9658d25cbf790f not found: ID does not exist" Aug 04 17:54:44.629236 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.629198 2572 scope.go:117] "RemoveContainer" containerID="30247426d51c290e26690504fa14b92ffd999777a3a6f515efe362698500260c" Aug 04 17:54:44.629385 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.629370 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30247426d51c290e26690504fa14b92ffd999777a3a6f515efe362698500260c"} err="failed to get container status \"30247426d51c290e26690504fa14b92ffd999777a3a6f515efe362698500260c\": rpc error: code = NotFound desc = could not find container \"30247426d51c290e26690504fa14b92ffd999777a3a6f515efe362698500260c\": container with ID starting with 30247426d51c290e26690504fa14b92ffd999777a3a6f515efe362698500260c not found: ID does not exist" Aug 04 17:54:44.662224 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.662190 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 17:54:44.662601 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.662582 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="kube-rbac-proxy" Aug 04 17:54:44.662726 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.662612 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="kube-rbac-proxy" Aug 04 17:54:44.662726 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.662629 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="config-reloader" Aug 04 17:54:44.662726 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.662637 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="config-reloader" Aug 04 17:54:44.662726 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.662647 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="prom-label-proxy" Aug 04 17:54:44.662726 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.662656 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="prom-label-proxy" Aug 04 17:54:44.662726 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.662686 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="kube-rbac-proxy-metric" Aug 04 17:54:44.662726 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.662696 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="kube-rbac-proxy-metric" Aug 04 17:54:44.662726 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.662709 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="alertmanager" Aug 04 17:54:44.662726 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.662717 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="alertmanager" Aug 04 17:54:44.663150 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.662738 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="init-config-reloader" Aug 04 17:54:44.663150 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.662746 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="init-config-reloader" Aug 04 17:54:44.663150 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.662758 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="kube-rbac-proxy-web" Aug 04 17:54:44.663150 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.662766 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="kube-rbac-proxy-web" Aug 04 17:54:44.663150 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.662844 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="alertmanager" Aug 04 17:54:44.663150 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.662857 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="kube-rbac-proxy-metric" Aug 04 17:54:44.663150 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.662868 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="prom-label-proxy" Aug 04 17:54:44.663150 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.662878 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="config-reloader" Aug 04 17:54:44.663150 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.662887 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="kube-rbac-proxy-web" Aug 04 17:54:44.663150 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.662896 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" containerName="kube-rbac-proxy" Aug 04 17:54:44.666544 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.666526 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.669248 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.669231 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Aug 04 17:54:44.675057 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.673846 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Aug 04 17:54:44.675057 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.674297 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Aug 04 17:54:44.675057 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.674693 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Aug 04 17:54:44.675057 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.674993 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Aug 04 17:54:44.675320 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.675257 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Aug 04 17:54:44.675450 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.675423 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/de29157d-9827-4529-ae90-6f457f90d7db-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.675533 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.675430 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Aug 04 17:54:44.675533 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.675471 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-c5x77\"" Aug 04 17:54:44.675533 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.675494 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de29157d-9827-4529-ae90-6f457f90d7db-config-out\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.675690 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.675566 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de29157d-9827-4529-ae90-6f457f90d7db-tls-assets\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.675690 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.675623 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de29157d-9827-4529-ae90-6f457f90d7db-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.675792 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.675689 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/de29157d-9827-4529-ae90-6f457f90d7db-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.675792 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.675726 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/de29157d-9827-4529-ae90-6f457f90d7db-config-volume\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.675792 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.675753 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/de29157d-9827-4529-ae90-6f457f90d7db-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.675907 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.675798 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Aug 04 17:54:44.675907 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.675813 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de29157d-9827-4529-ae90-6f457f90d7db-web-config\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.675907 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.675844 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de29157d-9827-4529-ae90-6f457f90d7db-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.675907 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.675893 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwb7k\" (UniqueName: \"kubernetes.io/projected/de29157d-9827-4529-ae90-6f457f90d7db-kube-api-access-fwb7k\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.676038 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.675917 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/de29157d-9827-4529-ae90-6f457f90d7db-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.676038 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.675933 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de29157d-9827-4529-ae90-6f457f90d7db-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.676038 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.675988 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de29157d-9827-4529-ae90-6f457f90d7db-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.682557 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.682536 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Aug 04 17:54:44.695727 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.695704 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 17:54:44.776616 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.776585 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de29157d-9827-4529-ae90-6f457f90d7db-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.776827 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.776624 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/de29157d-9827-4529-ae90-6f457f90d7db-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.776827 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.776644 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de29157d-9827-4529-ae90-6f457f90d7db-config-out\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.776827 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.776688 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de29157d-9827-4529-ae90-6f457f90d7db-tls-assets\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.776827 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.776718 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de29157d-9827-4529-ae90-6f457f90d7db-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.776827 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.776739 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/de29157d-9827-4529-ae90-6f457f90d7db-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.776827 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.776761 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/de29157d-9827-4529-ae90-6f457f90d7db-config-volume\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.776827 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.776783 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/de29157d-9827-4529-ae90-6f457f90d7db-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.776827 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.776810 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de29157d-9827-4529-ae90-6f457f90d7db-web-config\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.778081 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.778054 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de29157d-9827-4529-ae90-6f457f90d7db-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.778207 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.778094 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwb7k\" (UniqueName: \"kubernetes.io/projected/de29157d-9827-4529-ae90-6f457f90d7db-kube-api-access-fwb7k\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.778207 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.778126 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/de29157d-9827-4529-ae90-6f457f90d7db-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.778207 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.778155 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de29157d-9827-4529-ae90-6f457f90d7db-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.778538 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.778511 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de29157d-9827-4529-ae90-6f457f90d7db-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.779921 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.779896 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/de29157d-9827-4529-ae90-6f457f90d7db-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.780031 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.779909 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/de29157d-9827-4529-ae90-6f457f90d7db-web-config\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.780031 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.779965 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/de29157d-9827-4529-ae90-6f457f90d7db-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.780383 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.780284 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/de29157d-9827-4529-ae90-6f457f90d7db-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.780383 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.780324 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/de29157d-9827-4529-ae90-6f457f90d7db-config-out\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.780383 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.780326 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/de29157d-9827-4529-ae90-6f457f90d7db-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.780565 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.780486 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/de29157d-9827-4529-ae90-6f457f90d7db-tls-assets\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.780742 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.780718 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/de29157d-9827-4529-ae90-6f457f90d7db-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.780999 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.780844 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de29157d-9827-4529-ae90-6f457f90d7db-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.781933 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.781914 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/de29157d-9827-4529-ae90-6f457f90d7db-config-volume\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.782548 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.782525 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/de29157d-9827-4529-ae90-6f457f90d7db-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.794840 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.794823 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwb7k\" (UniqueName: \"kubernetes.io/projected/de29157d-9827-4529-ae90-6f457f90d7db-kube-api-access-fwb7k\") pod \"alertmanager-main-0\" (UID: \"de29157d-9827-4529-ae90-6f457f90d7db\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:44.819835 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.819814 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15a69893-7efd-4ce6-98da-d6bbe0a2ec3b" path="/var/lib/kubelet/pods/15a69893-7efd-4ce6-98da-d6bbe0a2ec3b/volumes" Aug 04 17:54:44.977577 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:44.977496 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 17:54:45.123889 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:45.123863 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 17:54:45.125981 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:54:45.125956 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde29157d_9827_4529_ae90_6f457f90d7db.slice/crio-b38bea2e3ee83712f87ea23f4b79724889039e45b24e6a359b5a418ab09073ca WatchSource:0}: Error finding container b38bea2e3ee83712f87ea23f4b79724889039e45b24e6a359b5a418ab09073ca: Status 404 returned error can't find the container with id b38bea2e3ee83712f87ea23f4b79724889039e45b24e6a359b5a418ab09073ca Aug 04 17:54:45.582558 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:45.582522 2572 generic.go:356] "Generic (PLEG): container finished" podID="de29157d-9827-4529-ae90-6f457f90d7db" containerID="94dee488448a0fba77ba3ded91ffa35db18d950a842f689b5930d23635b88cff" exitCode=0 Aug 04 17:54:45.582726 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:45.582565 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de29157d-9827-4529-ae90-6f457f90d7db","Type":"ContainerDied","Data":"94dee488448a0fba77ba3ded91ffa35db18d950a842f689b5930d23635b88cff"} Aug 04 17:54:45.582726 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:45.582632 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de29157d-9827-4529-ae90-6f457f90d7db","Type":"ContainerStarted","Data":"b38bea2e3ee83712f87ea23f4b79724889039e45b24e6a359b5a418ab09073ca"} Aug 04 17:54:46.589330 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:46.589283 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de29157d-9827-4529-ae90-6f457f90d7db","Type":"ContainerStarted","Data":"90a1ae97212db3820d38f1f50d7609d5d0f5e9ecb277df2c5f033657abe211cd"} Aug 04 17:54:46.589726 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:46.589372 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de29157d-9827-4529-ae90-6f457f90d7db","Type":"ContainerStarted","Data":"5ac651f3f2b01e18188eb511e5d81ae97fa089fd83d23175509afbabf88827a0"} Aug 04 17:54:46.589726 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:46.589414 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de29157d-9827-4529-ae90-6f457f90d7db","Type":"ContainerStarted","Data":"3f5da5f84f883d26628614ff1054fb778bf4ab7987c4ab3c8a9b013776976f10"} Aug 04 17:54:46.589726 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:46.589456 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de29157d-9827-4529-ae90-6f457f90d7db","Type":"ContainerStarted","Data":"55cdb7b4728f79d97e1ef3e886d06d9f70c5df7570165b0880985f7abe32f5b7"} Aug 04 17:54:46.589726 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:46.589490 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de29157d-9827-4529-ae90-6f457f90d7db","Type":"ContainerStarted","Data":"1e03062f3710d2da2bad2b102692eff8ec1d8a7c55d23c0d007fdc0b81414746"} Aug 04 17:54:46.589726 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:46.589523 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"de29157d-9827-4529-ae90-6f457f90d7db","Type":"ContainerStarted","Data":"3eb71508883dc0374f9ff2cda15ec5a32d6b4e22ff0d86fc92390a8ee5701d2d"} Aug 04 17:54:46.627640 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:46.627590 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.608671157 podStartE2EDuration="2.627573622s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.902465ms" isStatefulPod=false podCreationTimestamp="2026-08-04 17:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:54:46.625791031 +0000 UTC m=+142.316482082" watchObservedRunningTime="2026-08-04 17:54:46.627573622 +0000 UTC m=+142.318264671" Aug 04 17:54:47.109702 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.109650 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 04 17:54:47.110310 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.110272 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="prometheus" containerID="cri-o://e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173" gracePeriod=600 Aug 04 17:54:47.110420 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.110326 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19" gracePeriod=600 Aug 04 17:54:47.110420 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.110348 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="thanos-sidecar" containerID="cri-o://40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea" gracePeriod=600 Aug 04 17:54:47.110420 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.110392 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="kube-rbac-proxy" containerID="cri-o://d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7" gracePeriod=600 Aug 04 17:54:47.110420 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.110351 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="kube-rbac-proxy-web" containerID="cri-o://a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7" gracePeriod=600 Aug 04 17:54:47.110625 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.110367 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="config-reloader" containerID="cri-o://ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7" gracePeriod=600 Aug 04 17:54:47.452695 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.452656 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.501753 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.501730 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-kube-rbac-proxy\") pod \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " Aug 04 17:54:47.501964 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.501775 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-configmap-metrics-client-ca\") pod \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " Aug 04 17:54:47.501964 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.501802 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-configmap-kubelet-serving-ca-bundle\") pod \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " Aug 04 17:54:47.501964 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.501825 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-k8s-rulefiles-2\") pod \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " Aug 04 17:54:47.501964 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.501862 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-web-config\") pod \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " Aug 04 17:54:47.501964 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.501889 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-k8s-rulefiles-0\") pod \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " Aug 04 17:54:47.501964 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.501912 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-config\") pod \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " Aug 04 17:54:47.501964 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.501939 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-prometheus-k8s-tls\") pod \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " Aug 04 17:54:47.502314 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.501967 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-trusted-ca-bundle\") pod \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " Aug 04 17:54:47.502314 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.502018 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-k8s-rulefiles-1\") pod \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " Aug 04 17:54:47.502314 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.502067 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " Aug 04 17:54:47.502314 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.502109 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-tls-assets\") pod \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " Aug 04 17:54:47.502314 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.502142 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-k8s-db\") pod \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " Aug 04 17:54:47.502314 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.502167 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "2a0d35e9-7cc7-45c0-ba83-b581c25e172e" (UID: "2a0d35e9-7cc7-45c0-ba83-b581c25e172e"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:54:47.502314 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.502175 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-configmap-serving-certs-ca-bundle\") pod \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " Aug 04 17:54:47.502314 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.502233 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-metrics-client-certs\") pod \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " Aug 04 17:54:47.502314 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.502258 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvlpm\" (UniqueName: \"kubernetes.io/projected/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-kube-api-access-zvlpm\") pod \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " Aug 04 17:54:47.502314 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.502280 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-thanos-prometheus-http-client-file\") pod \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " Aug 04 17:54:47.502314 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.502311 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-config-out\") pod \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " Aug 04 17:54:47.502867 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.502371 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " Aug 04 17:54:47.502867 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.502417 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-grpc-tls\") pod \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\" (UID: \"2a0d35e9-7cc7-45c0-ba83-b581c25e172e\") " Aug 04 17:54:47.502867 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.502479 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "2a0d35e9-7cc7-45c0-ba83-b581c25e172e" (UID: "2a0d35e9-7cc7-45c0-ba83-b581c25e172e"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:54:47.502867 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.502687 2572 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-configmap-metrics-client-ca\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:47.502867 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.502708 2572 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:47.503111 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.502986 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "2a0d35e9-7cc7-45c0-ba83-b581c25e172e" (UID: "2a0d35e9-7cc7-45c0-ba83-b581c25e172e"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:54:47.503634 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.503191 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "2a0d35e9-7cc7-45c0-ba83-b581c25e172e" (UID: "2a0d35e9-7cc7-45c0-ba83-b581c25e172e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:54:47.504862 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.504833 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "2a0d35e9-7cc7-45c0-ba83-b581c25e172e" (UID: "2a0d35e9-7cc7-45c0-ba83-b581c25e172e"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:54:47.505151 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.505125 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "2a0d35e9-7cc7-45c0-ba83-b581c25e172e" (UID: "2a0d35e9-7cc7-45c0-ba83-b581c25e172e"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:54:47.505224 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.505175 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "2a0d35e9-7cc7-45c0-ba83-b581c25e172e" (UID: "2a0d35e9-7cc7-45c0-ba83-b581c25e172e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:54:47.505289 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.505272 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "2a0d35e9-7cc7-45c0-ba83-b581c25e172e" (UID: "2a0d35e9-7cc7-45c0-ba83-b581c25e172e"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:54:47.507103 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.505677 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "2a0d35e9-7cc7-45c0-ba83-b581c25e172e" (UID: "2a0d35e9-7cc7-45c0-ba83-b581c25e172e"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:54:47.507103 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.506165 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "2a0d35e9-7cc7-45c0-ba83-b581c25e172e" (UID: "2a0d35e9-7cc7-45c0-ba83-b581c25e172e"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:54:47.507103 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.506405 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "2a0d35e9-7cc7-45c0-ba83-b581c25e172e" (UID: "2a0d35e9-7cc7-45c0-ba83-b581c25e172e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 17:54:47.507103 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.507067 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "2a0d35e9-7cc7-45c0-ba83-b581c25e172e" (UID: "2a0d35e9-7cc7-45c0-ba83-b581c25e172e"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 17:54:47.509205 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.509168 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "2a0d35e9-7cc7-45c0-ba83-b581c25e172e" (UID: "2a0d35e9-7cc7-45c0-ba83-b581c25e172e"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:54:47.509339 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.509299 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "2a0d35e9-7cc7-45c0-ba83-b581c25e172e" (UID: "2a0d35e9-7cc7-45c0-ba83-b581c25e172e"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:54:47.509449 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.509424 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "2a0d35e9-7cc7-45c0-ba83-b581c25e172e" (UID: "2a0d35e9-7cc7-45c0-ba83-b581c25e172e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:54:47.509513 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.509448 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "2a0d35e9-7cc7-45c0-ba83-b581c25e172e" (UID: "2a0d35e9-7cc7-45c0-ba83-b581c25e172e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:54:47.509566 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.509522 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-config" (OuterVolumeSpecName: "config") pod "2a0d35e9-7cc7-45c0-ba83-b581c25e172e" (UID: "2a0d35e9-7cc7-45c0-ba83-b581c25e172e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:54:47.509776 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.509751 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-config-out" (OuterVolumeSpecName: "config-out") pod "2a0d35e9-7cc7-45c0-ba83-b581c25e172e" (UID: "2a0d35e9-7cc7-45c0-ba83-b581c25e172e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 17:54:47.509776 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.509764 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-kube-api-access-zvlpm" (OuterVolumeSpecName: "kube-api-access-zvlpm") pod "2a0d35e9-7cc7-45c0-ba83-b581c25e172e" (UID: "2a0d35e9-7cc7-45c0-ba83-b581c25e172e"). InnerVolumeSpecName "kube-api-access-zvlpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 17:54:47.519700 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.519678 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-web-config" (OuterVolumeSpecName: "web-config") pod "2a0d35e9-7cc7-45c0-ba83-b581c25e172e" (UID: "2a0d35e9-7cc7-45c0-ba83-b581c25e172e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 17:54:47.600143 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.600107 2572 generic.go:356] "Generic (PLEG): container finished" podID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerID="e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19" exitCode=0 Aug 04 17:54:47.600143 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.600133 2572 generic.go:356] "Generic (PLEG): container finished" podID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerID="d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7" exitCode=0 Aug 04 17:54:47.600143 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.600140 2572 generic.go:356] "Generic (PLEG): container finished" podID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerID="a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7" exitCode=0 Aug 04 17:54:47.600143 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.600146 2572 generic.go:356] "Generic (PLEG): container finished" podID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerID="40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea" exitCode=0 Aug 04 17:54:47.600143 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.600151 2572 generic.go:356] "Generic (PLEG): container finished" podID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerID="ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7" exitCode=0 Aug 04 17:54:47.600143 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.600156 2572 generic.go:356] "Generic (PLEG): container finished" podID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerID="e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173" exitCode=0 Aug 04 17:54:47.600872 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.600191 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.600872 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.600201 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a0d35e9-7cc7-45c0-ba83-b581c25e172e","Type":"ContainerDied","Data":"e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19"} Aug 04 17:54:47.600872 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.600315 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a0d35e9-7cc7-45c0-ba83-b581c25e172e","Type":"ContainerDied","Data":"d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7"} Aug 04 17:54:47.600872 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.600356 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a0d35e9-7cc7-45c0-ba83-b581c25e172e","Type":"ContainerDied","Data":"a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7"} Aug 04 17:54:47.600872 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.600392 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a0d35e9-7cc7-45c0-ba83-b581c25e172e","Type":"ContainerDied","Data":"40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea"} Aug 04 17:54:47.600872 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.600385 2572 scope.go:117] "RemoveContainer" containerID="e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19" Aug 04 17:54:47.600872 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.600432 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a0d35e9-7cc7-45c0-ba83-b581c25e172e","Type":"ContainerDied","Data":"ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7"} Aug 04 17:54:47.600872 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.600469 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a0d35e9-7cc7-45c0-ba83-b581c25e172e","Type":"ContainerDied","Data":"e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173"} Aug 04 17:54:47.600872 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.600504 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2a0d35e9-7cc7-45c0-ba83-b581c25e172e","Type":"ContainerDied","Data":"1ff13c4be5374ddd59786f1cec4ae81eac5a4b90a8a083d9401de89b5db6077b"} Aug 04 17:54:47.603235 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.603216 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:47.603235 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.603235 2572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:47.603376 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.603246 2572 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-tls-assets\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:47.603376 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.603255 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-k8s-db\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:47.603376 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.603263 2572 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-metrics-client-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:47.603376 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.603271 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvlpm\" (UniqueName: \"kubernetes.io/projected/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-kube-api-access-zvlpm\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:47.603376 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.603279 2572 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:47.603376 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.603288 2572 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-config-out\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:47.603376 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.603296 2572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:47.603376 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.603304 2572 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-grpc-tls\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:47.603376 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.603312 2572 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-kube-rbac-proxy\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:47.603376 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.603319 2572 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:47.603376 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.603329 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:47.603376 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.603353 2572 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-web-config\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:47.603376 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.603361 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:47.603376 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.603369 2572 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-config\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:47.603376 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.603377 2572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:47.603840 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.603385 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a0d35e9-7cc7-45c0-ba83-b581c25e172e-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 17:54:47.608626 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.608469 2572 scope.go:117] "RemoveContainer" containerID="d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7" Aug 04 17:54:47.616186 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.616162 2572 scope.go:117] "RemoveContainer" containerID="a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7" Aug 04 17:54:47.623133 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.623115 2572 scope.go:117] "RemoveContainer" containerID="40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea" Aug 04 17:54:47.629974 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.629957 2572 scope.go:117] "RemoveContainer" containerID="ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7" Aug 04 17:54:47.637584 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.637521 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 04 17:54:47.639007 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.638984 2572 scope.go:117] "RemoveContainer" containerID="e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173" Aug 04 17:54:47.643159 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.643137 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 04 17:54:47.647603 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.647579 2572 scope.go:117] "RemoveContainer" containerID="5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4" Aug 04 17:54:47.654223 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.654206 2572 scope.go:117] "RemoveContainer" containerID="e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19" Aug 04 17:54:47.654480 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:54:47.654463 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19\": container with ID starting with e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19 not found: ID does not exist" containerID="e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19" Aug 04 17:54:47.654538 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.654486 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19"} err="failed to get container status \"e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19\": rpc error: code = NotFound desc = could not find container \"e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19\": container with ID starting with e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19 not found: ID does not exist" Aug 04 17:54:47.654538 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.654505 2572 scope.go:117] "RemoveContainer" containerID="d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7" Aug 04 17:54:47.654724 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:54:47.654707 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7\": container with ID starting with d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7 not found: ID does not exist" containerID="d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7" Aug 04 17:54:47.654799 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.654725 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7"} err="failed to get container status \"d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7\": rpc error: code = NotFound desc = could not find container \"d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7\": container with ID starting with d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7 not found: ID does not exist" Aug 04 17:54:47.654799 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.654738 2572 scope.go:117] "RemoveContainer" containerID="a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7" Aug 04 17:54:47.654972 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:54:47.654956 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7\": container with ID starting with a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7 not found: ID does not exist" containerID="a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7" Aug 04 17:54:47.655009 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.654979 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7"} err="failed to get container status \"a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7\": rpc error: code = NotFound desc = could not find container \"a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7\": container with ID starting with a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7 not found: ID does not exist" Aug 04 17:54:47.655009 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.654991 2572 scope.go:117] "RemoveContainer" containerID="40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea" Aug 04 17:54:47.655249 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:54:47.655220 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea\": container with ID starting with 40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea not found: ID does not exist" containerID="40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea" Aug 04 17:54:47.655288 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.655254 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea"} err="failed to get container status \"40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea\": rpc error: code = NotFound desc = could not find container \"40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea\": container with ID starting with 40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea not found: ID does not exist" Aug 04 17:54:47.655288 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.655270 2572 scope.go:117] "RemoveContainer" containerID="ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7" Aug 04 17:54:47.655524 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:54:47.655505 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7\": container with ID starting with ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7 not found: ID does not exist" containerID="ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7" Aug 04 17:54:47.655560 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.655529 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7"} err="failed to get container status \"ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7\": rpc error: code = NotFound desc = could not find container \"ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7\": container with ID starting with ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7 not found: ID does not exist" Aug 04 17:54:47.655560 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.655543 2572 scope.go:117] "RemoveContainer" containerID="e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173" Aug 04 17:54:47.655794 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:54:47.655776 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173\": container with ID starting with e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173 not found: ID does not exist" containerID="e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173" Aug 04 17:54:47.655872 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.655796 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173"} err="failed to get container status \"e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173\": rpc error: code = NotFound desc = could not find container \"e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173\": container with ID starting with e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173 not found: ID does not exist" Aug 04 17:54:47.655872 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.655810 2572 scope.go:117] "RemoveContainer" containerID="5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4" Aug 04 17:54:47.656033 ip-10-0-137-173 kubenswrapper[2572]: E0804 17:54:47.656020 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4\": container with ID starting with 5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4 not found: ID does not exist" containerID="5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4" Aug 04 17:54:47.656076 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.656037 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4"} err="failed to get container status \"5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4\": rpc error: code = NotFound desc = could not find container \"5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4\": container with ID starting with 5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4 not found: ID does not exist" Aug 04 17:54:47.656076 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.656049 2572 scope.go:117] "RemoveContainer" containerID="e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19" Aug 04 17:54:47.656290 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.656270 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19"} err="failed to get container status \"e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19\": rpc error: code = NotFound desc = could not find container \"e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19\": container with ID starting with e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19 not found: ID does not exist" Aug 04 17:54:47.656334 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.656291 2572 scope.go:117] "RemoveContainer" containerID="d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7" Aug 04 17:54:47.656526 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.656509 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7"} err="failed to get container status \"d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7\": rpc error: code = NotFound desc = could not find container \"d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7\": container with ID starting with d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7 not found: ID does not exist" Aug 04 17:54:47.656572 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.656527 2572 scope.go:117] "RemoveContainer" containerID="a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7" Aug 04 17:54:47.656807 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.656789 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7"} err="failed to get container status \"a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7\": rpc error: code = NotFound desc = could not find container \"a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7\": container with ID starting with a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7 not found: ID does not exist" Aug 04 17:54:47.656856 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.656808 2572 scope.go:117] "RemoveContainer" containerID="40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea" Aug 04 17:54:47.657023 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.657005 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea"} err="failed to get container status \"40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea\": rpc error: code = NotFound desc = could not find container \"40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea\": container with ID starting with 40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea not found: ID does not exist" Aug 04 17:54:47.657089 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.657025 2572 scope.go:117] "RemoveContainer" containerID="ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7" Aug 04 17:54:47.657240 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.657224 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7"} err="failed to get container status \"ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7\": rpc error: code = NotFound desc = could not find container \"ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7\": container with ID starting with ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7 not found: ID does not exist" Aug 04 17:54:47.657288 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.657241 2572 scope.go:117] "RemoveContainer" containerID="e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173" Aug 04 17:54:47.657427 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.657409 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173"} err="failed to get container status \"e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173\": rpc error: code = NotFound desc = could not find container \"e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173\": container with ID starting with e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173 not found: ID does not exist" Aug 04 17:54:47.657427 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.657427 2572 scope.go:117] "RemoveContainer" containerID="5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4" Aug 04 17:54:47.657620 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.657603 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4"} err="failed to get container status \"5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4\": rpc error: code = NotFound desc = could not find container \"5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4\": container with ID starting with 5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4 not found: ID does not exist" Aug 04 17:54:47.657679 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.657621 2572 scope.go:117] "RemoveContainer" containerID="e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19" Aug 04 17:54:47.657832 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.657807 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19"} err="failed to get container status \"e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19\": rpc error: code = NotFound desc = could not find container \"e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19\": container with ID starting with e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19 not found: ID does not exist" Aug 04 17:54:47.657832 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.657825 2572 scope.go:117] "RemoveContainer" containerID="d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7" Aug 04 17:54:47.657989 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.657972 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7"} err="failed to get container status \"d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7\": rpc error: code = NotFound desc = could not find container \"d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7\": container with ID starting with d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7 not found: ID does not exist" Aug 04 17:54:47.657989 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.657987 2572 scope.go:117] "RemoveContainer" containerID="a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7" Aug 04 17:54:47.658165 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.658149 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7"} err="failed to get container status \"a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7\": rpc error: code = NotFound desc = could not find container \"a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7\": container with ID starting with a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7 not found: ID does not exist" Aug 04 17:54:47.658207 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.658164 2572 scope.go:117] "RemoveContainer" containerID="40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea" Aug 04 17:54:47.658331 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.658317 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea"} err="failed to get container status \"40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea\": rpc error: code = NotFound desc = could not find container \"40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea\": container with ID starting with 40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea not found: ID does not exist" Aug 04 17:54:47.658370 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.658331 2572 scope.go:117] "RemoveContainer" containerID="ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7" Aug 04 17:54:47.658503 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.658486 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7"} err="failed to get container status \"ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7\": rpc error: code = NotFound desc = could not find container \"ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7\": container with ID starting with ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7 not found: ID does not exist" Aug 04 17:54:47.658538 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.658504 2572 scope.go:117] "RemoveContainer" containerID="e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173" Aug 04 17:54:47.658709 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.658693 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173"} err="failed to get container status \"e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173\": rpc error: code = NotFound desc = could not find container \"e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173\": container with ID starting with e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173 not found: ID does not exist" Aug 04 17:54:47.658752 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.658710 2572 scope.go:117] "RemoveContainer" containerID="5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4" Aug 04 17:54:47.658901 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.658883 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4"} err="failed to get container status \"5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4\": rpc error: code = NotFound desc = could not find container \"5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4\": container with ID starting with 5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4 not found: ID does not exist" Aug 04 17:54:47.658967 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.658904 2572 scope.go:117] "RemoveContainer" containerID="e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19" Aug 04 17:54:47.659103 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.659088 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19"} err="failed to get container status \"e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19\": rpc error: code = NotFound desc = could not find container \"e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19\": container with ID starting with e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19 not found: ID does not exist" Aug 04 17:54:47.659151 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.659103 2572 scope.go:117] "RemoveContainer" containerID="d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7" Aug 04 17:54:47.659274 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.659260 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7"} err="failed to get container status \"d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7\": rpc error: code = NotFound desc = could not find container \"d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7\": container with ID starting with d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7 not found: ID does not exist" Aug 04 17:54:47.659321 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.659274 2572 scope.go:117] "RemoveContainer" containerID="a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7" Aug 04 17:54:47.659457 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.659440 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7"} err="failed to get container status \"a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7\": rpc error: code = NotFound desc = could not find container \"a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7\": container with ID starting with a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7 not found: ID does not exist" Aug 04 17:54:47.659521 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.659459 2572 scope.go:117] "RemoveContainer" containerID="40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea" Aug 04 17:54:47.659657 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.659642 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea"} err="failed to get container status \"40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea\": rpc error: code = NotFound desc = could not find container \"40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea\": container with ID starting with 40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea not found: ID does not exist" Aug 04 17:54:47.659732 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.659658 2572 scope.go:117] "RemoveContainer" containerID="ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7" Aug 04 17:54:47.659909 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.659891 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7"} err="failed to get container status \"ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7\": rpc error: code = NotFound desc = could not find container \"ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7\": container with ID starting with ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7 not found: ID does not exist" Aug 04 17:54:47.659963 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.659909 2572 scope.go:117] "RemoveContainer" containerID="e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173" Aug 04 17:54:47.660100 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.660086 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173"} err="failed to get container status \"e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173\": rpc error: code = NotFound desc = could not find container \"e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173\": container with ID starting with e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173 not found: ID does not exist" Aug 04 17:54:47.660142 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.660102 2572 scope.go:117] "RemoveContainer" containerID="5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4" Aug 04 17:54:47.660329 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.660310 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4"} err="failed to get container status \"5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4\": rpc error: code = NotFound desc = could not find container \"5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4\": container with ID starting with 5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4 not found: ID does not exist" Aug 04 17:54:47.660386 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.660329 2572 scope.go:117] "RemoveContainer" containerID="e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19" Aug 04 17:54:47.660498 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.660482 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19"} err="failed to get container status \"e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19\": rpc error: code = NotFound desc = could not find container \"e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19\": container with ID starting with e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19 not found: ID does not exist" Aug 04 17:54:47.660538 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.660501 2572 scope.go:117] "RemoveContainer" containerID="d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7" Aug 04 17:54:47.660700 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.660659 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7"} err="failed to get container status \"d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7\": rpc error: code = NotFound desc = could not find container \"d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7\": container with ID starting with d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7 not found: ID does not exist" Aug 04 17:54:47.660744 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.660702 2572 scope.go:117] "RemoveContainer" containerID="a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7" Aug 04 17:54:47.660883 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.660865 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7"} err="failed to get container status \"a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7\": rpc error: code = NotFound desc = could not find container \"a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7\": container with ID starting with a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7 not found: ID does not exist" Aug 04 17:54:47.660919 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.660884 2572 scope.go:117] "RemoveContainer" containerID="40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea" Aug 04 17:54:47.661100 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.661086 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea"} err="failed to get container status \"40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea\": rpc error: code = NotFound desc = could not find container \"40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea\": container with ID starting with 40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea not found: ID does not exist" Aug 04 17:54:47.661135 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.661100 2572 scope.go:117] "RemoveContainer" containerID="ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7" Aug 04 17:54:47.661316 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.661293 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7"} err="failed to get container status \"ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7\": rpc error: code = NotFound desc = could not find container \"ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7\": container with ID starting with ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7 not found: ID does not exist" Aug 04 17:54:47.661351 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.661317 2572 scope.go:117] "RemoveContainer" containerID="e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173" Aug 04 17:54:47.661511 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.661496 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173"} err="failed to get container status \"e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173\": rpc error: code = NotFound desc = could not find container \"e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173\": container with ID starting with e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173 not found: ID does not exist" Aug 04 17:54:47.661547 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.661512 2572 scope.go:117] "RemoveContainer" containerID="5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4" Aug 04 17:54:47.661741 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.661726 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4"} err="failed to get container status \"5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4\": rpc error: code = NotFound desc = could not find container \"5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4\": container with ID starting with 5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4 not found: ID does not exist" Aug 04 17:54:47.661777 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.661742 2572 scope.go:117] "RemoveContainer" containerID="e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19" Aug 04 17:54:47.661968 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.661953 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19"} err="failed to get container status \"e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19\": rpc error: code = NotFound desc = could not find container \"e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19\": container with ID starting with e88d812be9f4f9c6c941af40d50805081be5908c23c1b4a116e32c8045d60b19 not found: ID does not exist" Aug 04 17:54:47.662017 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.661968 2572 scope.go:117] "RemoveContainer" containerID="d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7" Aug 04 17:54:47.662176 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.662160 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7"} err="failed to get container status \"d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7\": rpc error: code = NotFound desc = could not find container \"d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7\": container with ID starting with d51655167717104cc1cd49529fb2a88f86ed980a2b2c5578bfd38330c8e4daa7 not found: ID does not exist" Aug 04 17:54:47.662216 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.662177 2572 scope.go:117] "RemoveContainer" containerID="a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7" Aug 04 17:54:47.662386 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.662367 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7"} err="failed to get container status \"a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7\": rpc error: code = NotFound desc = could not find container \"a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7\": container with ID starting with a85e42cc1c2e5a213b9a870605ff96e2dafc1ac4cf2a45377d9d01a555bc76c7 not found: ID does not exist" Aug 04 17:54:47.662436 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.662386 2572 scope.go:117] "RemoveContainer" containerID="40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea" Aug 04 17:54:47.662587 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.662570 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea"} err="failed to get container status \"40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea\": rpc error: code = NotFound desc = could not find container \"40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea\": container with ID starting with 40894a9e4687ef195576b34b033b937ccc64a2653195865fc5268c794a590eea not found: ID does not exist" Aug 04 17:54:47.662628 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.662587 2572 scope.go:117] "RemoveContainer" containerID="ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7" Aug 04 17:54:47.662803 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.662787 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7"} err="failed to get container status \"ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7\": rpc error: code = NotFound desc = could not find container \"ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7\": container with ID starting with ffb560ee81e0955ac8efd9241fd03af9b024d2be7f77612e2629f0afbcabb9a7 not found: ID does not exist" Aug 04 17:54:47.662857 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.662803 2572 scope.go:117] "RemoveContainer" containerID="e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173" Aug 04 17:54:47.663021 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.663005 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173"} err="failed to get container status \"e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173\": rpc error: code = NotFound desc = could not find container \"e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173\": container with ID starting with e3e7f7576c56f0c62a19a158f49b7ee4e939e2b9ce537cb0ddb016319e3dc173 not found: ID does not exist" Aug 04 17:54:47.663071 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.663021 2572 scope.go:117] "RemoveContainer" containerID="5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4" Aug 04 17:54:47.663271 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.663249 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4"} err="failed to get container status \"5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4\": rpc error: code = NotFound desc = could not find container \"5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4\": container with ID starting with 5573c7be623c7398b9184a4d1f547d577ea3da0eecc64ecfc490e240f1f4a9e4 not found: ID does not exist" Aug 04 17:54:47.676838 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.676814 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 04 17:54:47.677087 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.677071 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="thanos-sidecar" Aug 04 17:54:47.677087 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.677084 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="thanos-sidecar" Aug 04 17:54:47.677192 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.677094 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="kube-rbac-proxy" Aug 04 17:54:47.677192 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.677099 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="kube-rbac-proxy" Aug 04 17:54:47.677192 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.677108 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="init-config-reloader" Aug 04 17:54:47.677192 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.677114 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="init-config-reloader" Aug 04 17:54:47.677192 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.677121 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="kube-rbac-proxy-thanos" Aug 04 17:54:47.677192 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.677126 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="kube-rbac-proxy-thanos" Aug 04 17:54:47.677192 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.677133 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="prometheus" Aug 04 17:54:47.677192 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.677138 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="prometheus" Aug 04 17:54:47.677192 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.677143 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="config-reloader" Aug 04 17:54:47.677192 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.677148 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="config-reloader" Aug 04 17:54:47.677192 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.677153 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="kube-rbac-proxy-web" Aug 04 17:54:47.677192 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.677159 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="kube-rbac-proxy-web" Aug 04 17:54:47.677557 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.677207 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="kube-rbac-proxy" Aug 04 17:54:47.677557 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.677212 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="kube-rbac-proxy-thanos" Aug 04 17:54:47.677557 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.677219 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="prometheus" Aug 04 17:54:47.677557 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.677224 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="config-reloader" Aug 04 17:54:47.677557 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.677231 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="thanos-sidecar" Aug 04 17:54:47.677557 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.677236 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" containerName="kube-rbac-proxy-web" Aug 04 17:54:47.680765 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.680749 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.692058 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.692039 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Aug 04 17:54:47.692159 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.692112 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Aug 04 17:54:47.692217 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.692172 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Aug 04 17:54:47.692277 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.692255 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Aug 04 17:54:47.692402 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.692384 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Aug 04 17:54:47.692503 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.692489 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Aug 04 17:54:47.692879 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.692765 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Aug 04 17:54:47.692879 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.692781 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Aug 04 17:54:47.693149 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.692903 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Aug 04 17:54:47.693149 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.692992 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Aug 04 17:54:47.693149 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.693102 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-40k3ra6v8s84b\"" Aug 04 17:54:47.693299 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.693165 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-92kc8\"" Aug 04 17:54:47.693299 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.693247 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Aug 04 17:54:47.693416 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.693330 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Aug 04 17:54:47.694070 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.694056 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Aug 04 17:54:47.694734 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.694719 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Aug 04 17:54:47.699253 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.699237 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Aug 04 17:54:47.703913 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.703893 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/42aefbf1-664e-4883-9277-18bfc9174296-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.704962 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.703937 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.704962 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.703962 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.704962 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.704019 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/42aefbf1-664e-4883-9277-18bfc9174296-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.704962 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.704064 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.704962 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.704114 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-web-config\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.704962 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.704169 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbw46\" (UniqueName: \"kubernetes.io/projected/42aefbf1-664e-4883-9277-18bfc9174296-kube-api-access-bbw46\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.704962 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.704198 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.704962 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.704251 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-config\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.704962 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.704312 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42aefbf1-664e-4883-9277-18bfc9174296-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.704962 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.704376 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42aefbf1-664e-4883-9277-18bfc9174296-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.704962 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.704420 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/42aefbf1-664e-4883-9277-18bfc9174296-config-out\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.704962 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.704454 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42aefbf1-664e-4883-9277-18bfc9174296-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.704962 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.704475 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/42aefbf1-664e-4883-9277-18bfc9174296-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.704962 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.704531 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.704962 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.704567 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.704962 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.704613 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/42aefbf1-664e-4883-9277-18bfc9174296-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.705729 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.704637 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42aefbf1-664e-4883-9277-18bfc9174296-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.705729 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.704683 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/42aefbf1-664e-4883-9277-18bfc9174296-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.705729 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.704719 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.705729 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.705356 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 04 17:54:47.805770 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.805748 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42aefbf1-664e-4883-9277-18bfc9174296-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.805898 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.805784 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42aefbf1-664e-4883-9277-18bfc9174296-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.805898 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.805804 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/42aefbf1-664e-4883-9277-18bfc9174296-config-out\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.805898 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.805821 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42aefbf1-664e-4883-9277-18bfc9174296-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.805898 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.805838 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/42aefbf1-664e-4883-9277-18bfc9174296-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.805898 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.805865 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.805898 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.805891 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.806388 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.805917 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/42aefbf1-664e-4883-9277-18bfc9174296-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.806388 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.806071 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42aefbf1-664e-4883-9277-18bfc9174296-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.806388 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.806107 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/42aefbf1-664e-4883-9277-18bfc9174296-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.806388 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.806138 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.806388 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.806172 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/42aefbf1-664e-4883-9277-18bfc9174296-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.806388 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.806205 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.806388 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.806229 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.806388 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.806254 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/42aefbf1-664e-4883-9277-18bfc9174296-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.806388 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.806291 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.806388 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.806315 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-web-config\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.806388 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.806346 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbw46\" (UniqueName: \"kubernetes.io/projected/42aefbf1-664e-4883-9277-18bfc9174296-kube-api-access-bbw46\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.806388 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.806386 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.806993 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.806418 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-config\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.806993 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.806512 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42aefbf1-664e-4883-9277-18bfc9174296-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.806993 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.806586 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42aefbf1-664e-4883-9277-18bfc9174296-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.806993 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.806982 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/42aefbf1-664e-4883-9277-18bfc9174296-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.807183 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.807118 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42aefbf1-664e-4883-9277-18bfc9174296-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.807640 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.807584 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/42aefbf1-664e-4883-9277-18bfc9174296-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.809110 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.809083 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/42aefbf1-664e-4883-9277-18bfc9174296-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.810191 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.809751 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.810191 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.809852 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.810191 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.810042 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.810191 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.810153 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-config\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.810191 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.810184 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42aefbf1-664e-4883-9277-18bfc9174296-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.810731 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.810705 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/42aefbf1-664e-4883-9277-18bfc9174296-config-out\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.810825 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.810807 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-web-config\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.811032 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.811010 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.811385 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.811357 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.811547 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.811528 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.811619 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.811566 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/42aefbf1-664e-4883-9277-18bfc9174296-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.812216 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.812200 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/42aefbf1-664e-4883-9277-18bfc9174296-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.812459 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.812444 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/42aefbf1-664e-4883-9277-18bfc9174296-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.817821 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.817803 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbw46\" (UniqueName: \"kubernetes.io/projected/42aefbf1-664e-4883-9277-18bfc9174296-kube-api-access-bbw46\") pod \"prometheus-k8s-0\" (UID: \"42aefbf1-664e-4883-9277-18bfc9174296\") " pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:47.990721 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:47.990645 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:54:48.125614 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:48.125583 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Aug 04 17:54:48.128129 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:54:48.128106 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42aefbf1_664e_4883_9277_18bfc9174296.slice/crio-e19e89458b815a6d43d58cf0d248b0e1fc8a77a0f63e5c602f14653fdb68ec6f WatchSource:0}: Error finding container e19e89458b815a6d43d58cf0d248b0e1fc8a77a0f63e5c602f14653fdb68ec6f: Status 404 returned error can't find the container with id e19e89458b815a6d43d58cf0d248b0e1fc8a77a0f63e5c602f14653fdb68ec6f Aug 04 17:54:48.605306 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:48.605268 2572 generic.go:356] "Generic (PLEG): container finished" podID="42aefbf1-664e-4883-9277-18bfc9174296" containerID="acde512198556156c529aea2860f7f4b3f14a2d334be371f88b6ff6ac810b9d1" exitCode=0 Aug 04 17:54:48.605720 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:48.605351 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42aefbf1-664e-4883-9277-18bfc9174296","Type":"ContainerDied","Data":"acde512198556156c529aea2860f7f4b3f14a2d334be371f88b6ff6ac810b9d1"} Aug 04 17:54:48.605720 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:48.605434 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42aefbf1-664e-4883-9277-18bfc9174296","Type":"ContainerStarted","Data":"e19e89458b815a6d43d58cf0d248b0e1fc8a77a0f63e5c602f14653fdb68ec6f"} Aug 04 17:54:48.823154 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:48.823113 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a0d35e9-7cc7-45c0-ba83-b581c25e172e" path="/var/lib/kubelet/pods/2a0d35e9-7cc7-45c0-ba83-b581c25e172e/volumes" Aug 04 17:54:49.614497 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:49.614464 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42aefbf1-664e-4883-9277-18bfc9174296","Type":"ContainerStarted","Data":"b9020622bd2a003110d8745cf9749e26c85bf77102d832e2d718fc8053bed9ff"} Aug 04 17:54:49.614894 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:49.614526 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42aefbf1-664e-4883-9277-18bfc9174296","Type":"ContainerStarted","Data":"621eebbe333480cf6ca23cd2088d8e7a2165c6f77d316e1246f78c6df1b12d0f"} Aug 04 17:54:49.614894 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:49.614553 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42aefbf1-664e-4883-9277-18bfc9174296","Type":"ContainerStarted","Data":"0fc8fbc70868334435a9e5ec35900569fef6b3a1593937ea709d4636043d5d28"} Aug 04 17:54:49.614894 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:49.614578 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42aefbf1-664e-4883-9277-18bfc9174296","Type":"ContainerStarted","Data":"44c1aa1006b241b8e9d1b82c57694bc1be46f523557b5c000ff31cd83af46023"} Aug 04 17:54:49.614894 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:49.614600 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42aefbf1-664e-4883-9277-18bfc9174296","Type":"ContainerStarted","Data":"5396f85c9720d2fc6d931ec9c3ee69c6ba79d2dbd5f4e4519610045bc2aaa208"} Aug 04 17:54:49.614894 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:49.614624 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"42aefbf1-664e-4883-9277-18bfc9174296","Type":"ContainerStarted","Data":"487faf8fc00438253a18b767d618e31d34871a419f9e2dd25e4a75711e69b7c6"} Aug 04 17:54:49.647593 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:49.647544 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.6294900549999998 podStartE2EDuration="2.647526969s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.036914ms" isStatefulPod=false podCreationTimestamp="2026-08-04 17:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:54:49.64528713 +0000 UTC m=+145.335978185" watchObservedRunningTime="2026-08-04 17:54:49.647526969 +0000 UTC m=+145.338218019" Aug 04 17:54:51.555096 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:51.555056 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:54:52.990936 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:54:52.990903 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:55:02.619150 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:55:02.619111 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:55:02.706134 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:55:02.706105 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:55:02.881259 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:55:02.881182 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:55:14.224139 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:55:14.224103 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:55:23.740327 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:55:23.740289 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:55:23.800901 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:55:23.800868 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:55:33.870884 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:55:33.870845 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:55:47.991832 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:55:47.991798 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:55:48.007711 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:55:48.007682 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:55:48.843778 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:55:48.843749 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Aug 04 17:55:56.959409 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:55:56.959375 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:55:57.059922 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:55:57.059864 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:56:03.992176 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:56:03.992122 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:56:08.968402 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:56:08.968368 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:56:09.042271 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:56:09.042238 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:56:14.869731 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:56:14.869692 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:56:14.973004 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:56:14.972960 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:56:15.032997 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:56:15.032959 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:56:26.148056 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:56:26.148019 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:56:26.251095 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:56:26.251061 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:56:57.178617 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:56:57.178525 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:56:57.278425 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:56:57.278390 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:57:18.803407 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:18.803370 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:57:20.163824 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:20.163789 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:57:22.619492 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:22.619458 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:57:22.752864 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:22.752825 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:57:24.394001 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:24.393968 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:57:24.494429 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:24.494398 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:57:24.646265 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:24.646179 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:57:24.703265 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:24.703238 2572 kubelet.go:1648] "Image garbage collection succeeded" Aug 04 17:57:24.852193 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:24.852058 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:57:27.070713 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:27.070656 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:57:27.230810 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:27.230758 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:57:27.560677 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:27.560646 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-z48jr"] Aug 04 17:57:27.565746 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:27.564365 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z48jr" Aug 04 17:57:27.567329 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:27.567304 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Aug 04 17:57:27.568209 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:27.568192 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Aug 04 17:57:27.568292 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:27.568193 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-clxl9\"" Aug 04 17:57:27.575289 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:27.575266 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-z48jr"] Aug 04 17:57:27.647241 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:27.647197 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcfvp\" (UniqueName: \"kubernetes.io/projected/47e6e35c-6761-41fc-824e-cb6b274929ca-kube-api-access-bcfvp\") pod \"cert-manager-cainjector-559cdd8c87-z48jr\" (UID: \"47e6e35c-6761-41fc-824e-cb6b274929ca\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z48jr" Aug 04 17:57:27.647428 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:27.647296 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47e6e35c-6761-41fc-824e-cb6b274929ca-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-z48jr\" (UID: \"47e6e35c-6761-41fc-824e-cb6b274929ca\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z48jr" Aug 04 17:57:27.748321 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:27.748292 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47e6e35c-6761-41fc-824e-cb6b274929ca-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-z48jr\" (UID: \"47e6e35c-6761-41fc-824e-cb6b274929ca\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z48jr" Aug 04 17:57:27.748460 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:27.748342 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcfvp\" (UniqueName: \"kubernetes.io/projected/47e6e35c-6761-41fc-824e-cb6b274929ca-kube-api-access-bcfvp\") pod \"cert-manager-cainjector-559cdd8c87-z48jr\" (UID: \"47e6e35c-6761-41fc-824e-cb6b274929ca\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z48jr" Aug 04 17:57:27.757315 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:27.757292 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47e6e35c-6761-41fc-824e-cb6b274929ca-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-z48jr\" (UID: \"47e6e35c-6761-41fc-824e-cb6b274929ca\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z48jr" Aug 04 17:57:27.757440 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:27.757422 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcfvp\" (UniqueName: \"kubernetes.io/projected/47e6e35c-6761-41fc-824e-cb6b274929ca-kube-api-access-bcfvp\") pod \"cert-manager-cainjector-559cdd8c87-z48jr\" (UID: \"47e6e35c-6761-41fc-824e-cb6b274929ca\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z48jr" Aug 04 17:57:27.889182 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:27.889104 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z48jr" Aug 04 17:57:28.019958 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:28.019929 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-z48jr"] Aug 04 17:57:28.022678 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:57:28.022636 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47e6e35c_6761_41fc_824e_cb6b274929ca.slice/crio-a8c3051afb03c04ceb86e496bde304365b4da74f5ac89eb691ed99ccddc4d022 WatchSource:0}: Error finding container a8c3051afb03c04ceb86e496bde304365b4da74f5ac89eb691ed99ccddc4d022: Status 404 returned error can't find the container with id a8c3051afb03c04ceb86e496bde304365b4da74f5ac89eb691ed99ccddc4d022 Aug 04 17:57:28.024539 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:28.024526 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 17:57:28.182739 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:28.182675 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z48jr" event={"ID":"47e6e35c-6761-41fc-824e-cb6b274929ca","Type":"ContainerStarted","Data":"a8c3051afb03c04ceb86e496bde304365b4da74f5ac89eb691ed99ccddc4d022"} Aug 04 17:57:29.409071 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:29.409005 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:57:29.467023 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:29.466990 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:57:32.200865 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:32.200830 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z48jr" event={"ID":"47e6e35c-6761-41fc-824e-cb6b274929ca","Type":"ContainerStarted","Data":"2864432ce48430794f4d2fca362b32c27c29a750c112e88a56e4bb3aac748201"} Aug 04 17:57:32.232080 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:32.232035 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z48jr" podStartSLOduration=2.094635696 podStartE2EDuration="5.232019246s" totalImagesPullingTime="3.13738355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:57:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:57:32.220812151 +0000 UTC m=+307.911503203" watchObservedRunningTime="2026-08-04 17:57:32.232019246 +0000 UTC m=+307.922710297" Aug 04 17:57:33.613352 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:33.613317 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:57:33.697903 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:33.697872 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:57:34.716411 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:34.716376 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:57:37.131822 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:37.131785 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:57:37.240206 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:37.240158 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:57:42.708236 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:42.708202 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:57:42.809853 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:42.809820 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:57:44.408585 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:57:44.408549 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:00.434262 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:00.434225 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:00.536406 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:00.536375 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:01.115840 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.115809 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:01.432307 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.432200 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:01.522944 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.522909 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:01.598465 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.598435 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz"] Aug 04 17:58:01.601846 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.601829 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz" Aug 04 17:58:01.606113 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.606092 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 04 17:58:01.606228 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.606115 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-w7vjj\"" Aug 04 17:58:01.607223 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.607206 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 04 17:58:01.607638 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.607623 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 04 17:58:01.608486 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.608462 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 04 17:58:01.611746 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.611730 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fdc80a1-1524-40c4-8e22-edfb5b0167ce-tmp\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz\" (UID: \"5fdc80a1-1524-40c4-8e22-edfb5b0167ce\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz" Aug 04 17:58:01.611834 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.611765 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5fdc80a1-1524-40c4-8e22-edfb5b0167ce-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz\" (UID: \"5fdc80a1-1524-40c4-8e22-edfb5b0167ce\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz" Aug 04 17:58:01.611834 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.611792 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pphsn\" (UniqueName: \"kubernetes.io/projected/5fdc80a1-1524-40c4-8e22-edfb5b0167ce-kube-api-access-pphsn\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz\" (UID: \"5fdc80a1-1524-40c4-8e22-edfb5b0167ce\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz" Aug 04 17:58:01.611907 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.611841 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5fdc80a1-1524-40c4-8e22-edfb5b0167ce-manifests\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz\" (UID: \"5fdc80a1-1524-40c4-8e22-edfb5b0167ce\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz" Aug 04 17:58:01.611907 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.611876 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5fdc80a1-1524-40c4-8e22-edfb5b0167ce-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz\" (UID: \"5fdc80a1-1524-40c4-8e22-edfb5b0167ce\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz" Aug 04 17:58:01.636246 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.636222 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz"] Aug 04 17:58:01.712405 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.712326 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fdc80a1-1524-40c4-8e22-edfb5b0167ce-tmp\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz\" (UID: \"5fdc80a1-1524-40c4-8e22-edfb5b0167ce\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz" Aug 04 17:58:01.712405 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.712374 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5fdc80a1-1524-40c4-8e22-edfb5b0167ce-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz\" (UID: \"5fdc80a1-1524-40c4-8e22-edfb5b0167ce\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz" Aug 04 17:58:01.712729 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.712411 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pphsn\" (UniqueName: \"kubernetes.io/projected/5fdc80a1-1524-40c4-8e22-edfb5b0167ce-kube-api-access-pphsn\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz\" (UID: \"5fdc80a1-1524-40c4-8e22-edfb5b0167ce\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz" Aug 04 17:58:01.712729 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.712429 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5fdc80a1-1524-40c4-8e22-edfb5b0167ce-manifests\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz\" (UID: \"5fdc80a1-1524-40c4-8e22-edfb5b0167ce\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz" Aug 04 17:58:01.712729 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.712450 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5fdc80a1-1524-40c4-8e22-edfb5b0167ce-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz\" (UID: \"5fdc80a1-1524-40c4-8e22-edfb5b0167ce\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz" Aug 04 17:58:01.712896 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.712866 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5fdc80a1-1524-40c4-8e22-edfb5b0167ce-manifests\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz\" (UID: \"5fdc80a1-1524-40c4-8e22-edfb5b0167ce\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz" Aug 04 17:58:01.714794 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.714772 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fdc80a1-1524-40c4-8e22-edfb5b0167ce-tmp\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz\" (UID: \"5fdc80a1-1524-40c4-8e22-edfb5b0167ce\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz" Aug 04 17:58:01.715017 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.714998 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5fdc80a1-1524-40c4-8e22-edfb5b0167ce-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz\" (UID: \"5fdc80a1-1524-40c4-8e22-edfb5b0167ce\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz" Aug 04 17:58:01.715080 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.715032 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5fdc80a1-1524-40c4-8e22-edfb5b0167ce-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz\" (UID: \"5fdc80a1-1524-40c4-8e22-edfb5b0167ce\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz" Aug 04 17:58:01.738212 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.738183 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pphsn\" (UniqueName: \"kubernetes.io/projected/5fdc80a1-1524-40c4-8e22-edfb5b0167ce-kube-api-access-pphsn\") pod \"opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz\" (UID: \"5fdc80a1-1524-40c4-8e22-edfb5b0167ce\") " pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz" Aug 04 17:58:01.911118 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:01.911076 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz" Aug 04 17:58:02.044597 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:02.044570 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz"] Aug 04 17:58:02.047173 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:58:02.047148 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fdc80a1_1524_40c4_8e22_edfb5b0167ce.slice/crio-2a7cc25598296d08506141c8f457c3e5352650df3449ba3b0dfdc70929f6f435 WatchSource:0}: Error finding container 2a7cc25598296d08506141c8f457c3e5352650df3449ba3b0dfdc70929f6f435: Status 404 returned error can't find the container with id 2a7cc25598296d08506141c8f457c3e5352650df3449ba3b0dfdc70929f6f435 Aug 04 17:58:02.163578 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:02.163550 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:02.311716 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:02.311684 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz" event={"ID":"5fdc80a1-1524-40c4-8e22-edfb5b0167ce","Type":"ContainerStarted","Data":"2a7cc25598296d08506141c8f457c3e5352650df3449ba3b0dfdc70929f6f435"} Aug 04 17:58:02.550851 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:02.550809 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:04.919766 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:04.919651 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:05.019878 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:05.019845 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:05.325797 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:05.325764 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz" event={"ID":"5fdc80a1-1524-40c4-8e22-edfb5b0167ce","Type":"ContainerStarted","Data":"c1fd032f9980e9b0fc7986cb51f35763ab18d5d30c014d94731681977dcba8e5"} Aug 04 17:58:06.331005 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:06.330972 2572 generic.go:356] "Generic (PLEG): container finished" podID="5fdc80a1-1524-40c4-8e22-edfb5b0167ce" containerID="c1fd032f9980e9b0fc7986cb51f35763ab18d5d30c014d94731681977dcba8e5" exitCode=0 Aug 04 17:58:06.331379 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:06.331071 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz" event={"ID":"5fdc80a1-1524-40c4-8e22-edfb5b0167ce","Type":"ContainerDied","Data":"c1fd032f9980e9b0fc7986cb51f35763ab18d5d30c014d94731681977dcba8e5"} Aug 04 17:58:07.336877 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:07.336841 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz" event={"ID":"5fdc80a1-1524-40c4-8e22-edfb5b0167ce","Type":"ContainerStarted","Data":"5d8087f021d2fbb28b22b9aa55cf239dd44f9f6932caf83b61f6636e9d4cb67a"} Aug 04 17:58:07.337258 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:07.336978 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz" Aug 04 17:58:18.342316 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:18.342237 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz" Aug 04 17:58:18.372479 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:18.372433 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz" podStartSLOduration=12.809350372 podStartE2EDuration="17.37241773s" totalImagesPullingTime="3.528935414s" totalInitContainerRuntime="1.034131944s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:58:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:58:07.390748459 +0000 UTC m=+343.081439545" watchObservedRunningTime="2026-08-04 17:58:18.37241773 +0000 UTC m=+354.063108781" Aug 04 17:58:20.866897 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:20.866855 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:20.977853 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:20.977805 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:21.562413 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:21.562342 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:21.670490 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:21.670453 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:21.756136 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:21.756103 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-798d66f497-gw6lc"] Aug 04 17:58:21.764239 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:21.764206 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-798d66f497-gw6lc" Aug 04 17:58:21.766990 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:21.766967 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Aug 04 17:58:21.767896 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:21.767872 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 04 17:58:21.768015 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:21.767971 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 04 17:58:21.768093 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:21.767902 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Aug 04 17:58:21.768174 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:21.767934 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-td6qp\"" Aug 04 17:58:21.777336 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:21.777316 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-798d66f497-gw6lc"] Aug 04 17:58:21.794260 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:21.794236 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:21.875281 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:21.875196 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4907cbad-4833-473c-9b6b-b41272f3c90a-tls-certs\") pod \"kube-auth-proxy-798d66f497-gw6lc\" (UID: \"4907cbad-4833-473c-9b6b-b41272f3c90a\") " pod="openshift-ingress/kube-auth-proxy-798d66f497-gw6lc" Aug 04 17:58:21.875281 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:21.875253 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68wgv\" (UniqueName: \"kubernetes.io/projected/4907cbad-4833-473c-9b6b-b41272f3c90a-kube-api-access-68wgv\") pod \"kube-auth-proxy-798d66f497-gw6lc\" (UID: \"4907cbad-4833-473c-9b6b-b41272f3c90a\") " pod="openshift-ingress/kube-auth-proxy-798d66f497-gw6lc" Aug 04 17:58:21.875712 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:21.875312 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4907cbad-4833-473c-9b6b-b41272f3c90a-tmp\") pod \"kube-auth-proxy-798d66f497-gw6lc\" (UID: \"4907cbad-4833-473c-9b6b-b41272f3c90a\") " pod="openshift-ingress/kube-auth-proxy-798d66f497-gw6lc" Aug 04 17:58:21.940067 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:21.940029 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:21.976222 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:21.976189 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4907cbad-4833-473c-9b6b-b41272f3c90a-tls-certs\") pod \"kube-auth-proxy-798d66f497-gw6lc\" (UID: \"4907cbad-4833-473c-9b6b-b41272f3c90a\") " pod="openshift-ingress/kube-auth-proxy-798d66f497-gw6lc" Aug 04 17:58:21.976374 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:21.976235 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68wgv\" (UniqueName: \"kubernetes.io/projected/4907cbad-4833-473c-9b6b-b41272f3c90a-kube-api-access-68wgv\") pod \"kube-auth-proxy-798d66f497-gw6lc\" (UID: \"4907cbad-4833-473c-9b6b-b41272f3c90a\") " pod="openshift-ingress/kube-auth-proxy-798d66f497-gw6lc" Aug 04 17:58:21.976374 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:21.976273 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4907cbad-4833-473c-9b6b-b41272f3c90a-tmp\") pod \"kube-auth-proxy-798d66f497-gw6lc\" (UID: \"4907cbad-4833-473c-9b6b-b41272f3c90a\") " pod="openshift-ingress/kube-auth-proxy-798d66f497-gw6lc" Aug 04 17:58:21.978653 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:21.978627 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4907cbad-4833-473c-9b6b-b41272f3c90a-tmp\") pod \"kube-auth-proxy-798d66f497-gw6lc\" (UID: \"4907cbad-4833-473c-9b6b-b41272f3c90a\") " pod="openshift-ingress/kube-auth-proxy-798d66f497-gw6lc" Aug 04 17:58:21.978653 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:21.978655 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4907cbad-4833-473c-9b6b-b41272f3c90a-tls-certs\") pod \"kube-auth-proxy-798d66f497-gw6lc\" (UID: \"4907cbad-4833-473c-9b6b-b41272f3c90a\") " pod="openshift-ingress/kube-auth-proxy-798d66f497-gw6lc" Aug 04 17:58:21.987657 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:21.987637 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68wgv\" (UniqueName: \"kubernetes.io/projected/4907cbad-4833-473c-9b6b-b41272f3c90a-kube-api-access-68wgv\") pod \"kube-auth-proxy-798d66f497-gw6lc\" (UID: \"4907cbad-4833-473c-9b6b-b41272f3c90a\") " pod="openshift-ingress/kube-auth-proxy-798d66f497-gw6lc" Aug 04 17:58:22.004772 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:22.004734 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:22.075926 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:22.075889 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-798d66f497-gw6lc" Aug 04 17:58:22.106333 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:22.106305 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:22.222971 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:22.222941 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-798d66f497-gw6lc"] Aug 04 17:58:22.226199 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:58:22.226173 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4907cbad_4833_473c_9b6b_b41272f3c90a.slice/crio-6302f89f4b67d4f72646964bc3e2e9f2a0a121d8385c69a1e4d191f5fa9aa72e WatchSource:0}: Error finding container 6302f89f4b67d4f72646964bc3e2e9f2a0a121d8385c69a1e4d191f5fa9aa72e: Status 404 returned error can't find the container with id 6302f89f4b67d4f72646964bc3e2e9f2a0a121d8385c69a1e4d191f5fa9aa72e Aug 04 17:58:22.395124 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:22.395029 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-798d66f497-gw6lc" event={"ID":"4907cbad-4833-473c-9b6b-b41272f3c90a","Type":"ContainerStarted","Data":"6302f89f4b67d4f72646964bc3e2e9f2a0a121d8385c69a1e4d191f5fa9aa72e"} Aug 04 17:58:22.670899 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:22.670800 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:22.893196 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:22.893102 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:23.421123 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:23.421090 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:23.508765 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:23.508724 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:24.675129 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:24.675071 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:25.128332 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:25.128310 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Aug 04 17:58:25.409825 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:25.409735 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-798d66f497-gw6lc" event={"ID":"4907cbad-4833-473c-9b6b-b41272f3c90a","Type":"ContainerStarted","Data":"11dcbfb469de02be9dd8b35fdfd9e8bc3e4c7dfead2bf8323e7f4fc500e06e58"} Aug 04 17:58:25.432527 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:25.432479 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-798d66f497-gw6lc" podStartSLOduration=1.535445497 podStartE2EDuration="4.43246194s" totalImagesPullingTime="2.897016443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:58:25.429978178 +0000 UTC m=+361.120669229" watchObservedRunningTime="2026-08-04 17:58:25.43246194 +0000 UTC m=+361.123152991" Aug 04 17:58:27.386062 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:27.386024 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:27.489490 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:27.489453 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:27.588309 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:27.588278 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:27.650871 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:27.650800 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:29.870352 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:29.870319 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f"] Aug 04 17:58:29.873858 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:29.873843 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:29.884924 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:29.884908 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Aug 04 17:58:29.885347 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:29.885327 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Aug 04 17:58:29.885415 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:29.885329 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Aug 04 17:58:29.886466 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:29.886452 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wqsxr\"" Aug 04 17:58:29.887145 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:29.887132 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 04 17:58:29.924748 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:29.924718 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f"] Aug 04 17:58:29.947420 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:29.947389 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e191e8f4-ac16-4f98-8c33-0b9a6b313992-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-k5r8f\" (UID: \"e191e8f4-ac16-4f98-8c33-0b9a6b313992\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:29.947544 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:29.947426 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e191e8f4-ac16-4f98-8c33-0b9a6b313992-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-k5r8f\" (UID: \"e191e8f4-ac16-4f98-8c33-0b9a6b313992\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:29.947544 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:29.947451 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e191e8f4-ac16-4f98-8c33-0b9a6b313992-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-k5r8f\" (UID: \"e191e8f4-ac16-4f98-8c33-0b9a6b313992\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:29.947544 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:29.947500 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e191e8f4-ac16-4f98-8c33-0b9a6b313992-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-k5r8f\" (UID: \"e191e8f4-ac16-4f98-8c33-0b9a6b313992\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:29.947649 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:29.947541 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhrzn\" (UniqueName: \"kubernetes.io/projected/e191e8f4-ac16-4f98-8c33-0b9a6b313992-kube-api-access-xhrzn\") pod \"istiod-openshift-gateway-5677ccb49b-k5r8f\" (UID: \"e191e8f4-ac16-4f98-8c33-0b9a6b313992\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:29.947649 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:29.947594 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e191e8f4-ac16-4f98-8c33-0b9a6b313992-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-k5r8f\" (UID: \"e191e8f4-ac16-4f98-8c33-0b9a6b313992\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:29.947649 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:29.947635 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e191e8f4-ac16-4f98-8c33-0b9a6b313992-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-k5r8f\" (UID: \"e191e8f4-ac16-4f98-8c33-0b9a6b313992\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:30.048469 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:30.048419 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhrzn\" (UniqueName: \"kubernetes.io/projected/e191e8f4-ac16-4f98-8c33-0b9a6b313992-kube-api-access-xhrzn\") pod \"istiod-openshift-gateway-5677ccb49b-k5r8f\" (UID: \"e191e8f4-ac16-4f98-8c33-0b9a6b313992\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:30.048637 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:30.048491 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e191e8f4-ac16-4f98-8c33-0b9a6b313992-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-k5r8f\" (UID: \"e191e8f4-ac16-4f98-8c33-0b9a6b313992\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:30.048637 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:30.048530 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e191e8f4-ac16-4f98-8c33-0b9a6b313992-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-k5r8f\" (UID: \"e191e8f4-ac16-4f98-8c33-0b9a6b313992\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:30.048637 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:30.048584 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e191e8f4-ac16-4f98-8c33-0b9a6b313992-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-k5r8f\" (UID: \"e191e8f4-ac16-4f98-8c33-0b9a6b313992\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:30.048637 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:30.048615 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e191e8f4-ac16-4f98-8c33-0b9a6b313992-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-k5r8f\" (UID: \"e191e8f4-ac16-4f98-8c33-0b9a6b313992\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:30.048850 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:30.048639 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e191e8f4-ac16-4f98-8c33-0b9a6b313992-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-k5r8f\" (UID: \"e191e8f4-ac16-4f98-8c33-0b9a6b313992\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:30.048850 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:30.048693 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e191e8f4-ac16-4f98-8c33-0b9a6b313992-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-k5r8f\" (UID: \"e191e8f4-ac16-4f98-8c33-0b9a6b313992\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:30.049426 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:30.049285 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e191e8f4-ac16-4f98-8c33-0b9a6b313992-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-k5r8f\" (UID: \"e191e8f4-ac16-4f98-8c33-0b9a6b313992\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:30.051066 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:30.051040 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e191e8f4-ac16-4f98-8c33-0b9a6b313992-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-k5r8f\" (UID: \"e191e8f4-ac16-4f98-8c33-0b9a6b313992\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:30.051173 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:30.051132 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e191e8f4-ac16-4f98-8c33-0b9a6b313992-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-k5r8f\" (UID: \"e191e8f4-ac16-4f98-8c33-0b9a6b313992\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:30.051173 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:30.051164 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e191e8f4-ac16-4f98-8c33-0b9a6b313992-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-k5r8f\" (UID: \"e191e8f4-ac16-4f98-8c33-0b9a6b313992\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:30.051280 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:30.051223 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e191e8f4-ac16-4f98-8c33-0b9a6b313992-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-k5r8f\" (UID: \"e191e8f4-ac16-4f98-8c33-0b9a6b313992\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:30.066383 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:30.066362 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhrzn\" (UniqueName: \"kubernetes.io/projected/e191e8f4-ac16-4f98-8c33-0b9a6b313992-kube-api-access-xhrzn\") pod \"istiod-openshift-gateway-5677ccb49b-k5r8f\" (UID: \"e191e8f4-ac16-4f98-8c33-0b9a6b313992\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:30.074789 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:30.074762 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e191e8f4-ac16-4f98-8c33-0b9a6b313992-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-k5r8f\" (UID: \"e191e8f4-ac16-4f98-8c33-0b9a6b313992\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:30.182742 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:30.182648 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:30.360797 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:30.360766 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f"] Aug 04 17:58:30.364370 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:58:30.364343 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode191e8f4_ac16_4f98_8c33_0b9a6b313992.slice/crio-1cac46ec4bfb11295ec0dd4e877e48beebfe623266e68375135778e9ef50b1c9 WatchSource:0}: Error finding container 1cac46ec4bfb11295ec0dd4e877e48beebfe623266e68375135778e9ef50b1c9: Status 404 returned error can't find the container with id 1cac46ec4bfb11295ec0dd4e877e48beebfe623266e68375135778e9ef50b1c9 Aug 04 17:58:30.432272 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:30.432232 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" event={"ID":"e191e8f4-ac16-4f98-8c33-0b9a6b313992","Type":"ContainerStarted","Data":"1cac46ec4bfb11295ec0dd4e877e48beebfe623266e68375135778e9ef50b1c9"} Aug 04 17:58:31.271777 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:31.271743 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-55bf5c9f58-bfs22"] Aug 04 17:58:31.276269 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:31.276233 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-55bf5c9f58-bfs22" Aug 04 17:58:31.283038 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:31.283010 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Aug 04 17:58:31.283170 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:31.283088 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cbt2s\"" Aug 04 17:58:31.283242 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:31.283010 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Aug 04 17:58:31.283595 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:31.283575 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 04 17:58:31.283879 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:31.283859 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Aug 04 17:58:31.284213 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:31.284197 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 04 17:58:31.299687 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:31.298918 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-55bf5c9f58-bfs22"] Aug 04 17:58:31.360516 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:31.360479 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bb7ec098-1dd9-4212-bbf2-d69562f92b97-manager-config\") pod \"lws-controller-manager-55bf5c9f58-bfs22\" (UID: \"bb7ec098-1dd9-4212-bbf2-d69562f92b97\") " pod="openshift-lws-operator/lws-controller-manager-55bf5c9f58-bfs22" Aug 04 17:58:31.360863 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:31.360543 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb7ec098-1dd9-4212-bbf2-d69562f92b97-cert\") pod \"lws-controller-manager-55bf5c9f58-bfs22\" (UID: \"bb7ec098-1dd9-4212-bbf2-d69562f92b97\") " pod="openshift-lws-operator/lws-controller-manager-55bf5c9f58-bfs22" Aug 04 17:58:31.360863 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:31.360602 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb7ec098-1dd9-4212-bbf2-d69562f92b97-metrics-cert\") pod \"lws-controller-manager-55bf5c9f58-bfs22\" (UID: \"bb7ec098-1dd9-4212-bbf2-d69562f92b97\") " pod="openshift-lws-operator/lws-controller-manager-55bf5c9f58-bfs22" Aug 04 17:58:31.360863 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:31.360695 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjvb2\" (UniqueName: \"kubernetes.io/projected/bb7ec098-1dd9-4212-bbf2-d69562f92b97-kube-api-access-vjvb2\") pod \"lws-controller-manager-55bf5c9f58-bfs22\" (UID: \"bb7ec098-1dd9-4212-bbf2-d69562f92b97\") " pod="openshift-lws-operator/lws-controller-manager-55bf5c9f58-bfs22" Aug 04 17:58:31.461786 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:31.461653 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjvb2\" (UniqueName: \"kubernetes.io/projected/bb7ec098-1dd9-4212-bbf2-d69562f92b97-kube-api-access-vjvb2\") pod \"lws-controller-manager-55bf5c9f58-bfs22\" (UID: \"bb7ec098-1dd9-4212-bbf2-d69562f92b97\") " pod="openshift-lws-operator/lws-controller-manager-55bf5c9f58-bfs22" Aug 04 17:58:31.461786 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:31.461738 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bb7ec098-1dd9-4212-bbf2-d69562f92b97-manager-config\") pod \"lws-controller-manager-55bf5c9f58-bfs22\" (UID: \"bb7ec098-1dd9-4212-bbf2-d69562f92b97\") " pod="openshift-lws-operator/lws-controller-manager-55bf5c9f58-bfs22" Aug 04 17:58:31.462099 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:31.461838 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb7ec098-1dd9-4212-bbf2-d69562f92b97-cert\") pod \"lws-controller-manager-55bf5c9f58-bfs22\" (UID: \"bb7ec098-1dd9-4212-bbf2-d69562f92b97\") " pod="openshift-lws-operator/lws-controller-manager-55bf5c9f58-bfs22" Aug 04 17:58:31.462099 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:31.461903 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb7ec098-1dd9-4212-bbf2-d69562f92b97-metrics-cert\") pod \"lws-controller-manager-55bf5c9f58-bfs22\" (UID: \"bb7ec098-1dd9-4212-bbf2-d69562f92b97\") " pod="openshift-lws-operator/lws-controller-manager-55bf5c9f58-bfs22" Aug 04 17:58:31.462854 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:31.462828 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bb7ec098-1dd9-4212-bbf2-d69562f92b97-manager-config\") pod \"lws-controller-manager-55bf5c9f58-bfs22\" (UID: \"bb7ec098-1dd9-4212-bbf2-d69562f92b97\") " pod="openshift-lws-operator/lws-controller-manager-55bf5c9f58-bfs22" Aug 04 17:58:31.466630 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:31.466593 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb7ec098-1dd9-4212-bbf2-d69562f92b97-metrics-cert\") pod \"lws-controller-manager-55bf5c9f58-bfs22\" (UID: \"bb7ec098-1dd9-4212-bbf2-d69562f92b97\") " pod="openshift-lws-operator/lws-controller-manager-55bf5c9f58-bfs22" Aug 04 17:58:31.466765 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:31.466596 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb7ec098-1dd9-4212-bbf2-d69562f92b97-cert\") pod \"lws-controller-manager-55bf5c9f58-bfs22\" (UID: \"bb7ec098-1dd9-4212-bbf2-d69562f92b97\") " pod="openshift-lws-operator/lws-controller-manager-55bf5c9f58-bfs22" Aug 04 17:58:31.479615 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:31.479587 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjvb2\" (UniqueName: \"kubernetes.io/projected/bb7ec098-1dd9-4212-bbf2-d69562f92b97-kube-api-access-vjvb2\") pod \"lws-controller-manager-55bf5c9f58-bfs22\" (UID: \"bb7ec098-1dd9-4212-bbf2-d69562f92b97\") " pod="openshift-lws-operator/lws-controller-manager-55bf5c9f58-bfs22" Aug 04 17:58:31.592613 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:31.592576 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-55bf5c9f58-bfs22" Aug 04 17:58:31.837788 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:31.837761 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-55bf5c9f58-bfs22"] Aug 04 17:58:31.840024 ip-10-0-137-173 kubenswrapper[2572]: W0804 17:58:31.839994 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7ec098_1dd9_4212_bbf2_d69562f92b97.slice/crio-9ddb312bfc29067240da8add8540ca50d0749a42ce318403f7945dce70cb7932 WatchSource:0}: Error finding container 9ddb312bfc29067240da8add8540ca50d0749a42ce318403f7945dce70cb7932: Status 404 returned error can't find the container with id 9ddb312bfc29067240da8add8540ca50d0749a42ce318403f7945dce70cb7932 Aug 04 17:58:32.109158 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:32.109075 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:32.242459 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:32.242430 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:32.444488 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:32.444402 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-55bf5c9f58-bfs22" event={"ID":"bb7ec098-1dd9-4212-bbf2-d69562f92b97","Type":"ContainerStarted","Data":"9ddb312bfc29067240da8add8540ca50d0749a42ce318403f7945dce70cb7932"} Aug 04 17:58:33.040296 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:33.040256 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 17:58:33.040386 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:33.040344 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 17:58:33.452983 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:33.452947 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" event={"ID":"e191e8f4-ac16-4f98-8c33-0b9a6b313992","Type":"ContainerStarted","Data":"08127100f4b867ae3c0f6946b9c1832f2f994e3d47f8e38c5286efeab303802f"} Aug 04 17:58:33.453420 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:33.453172 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:33.571682 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:33.571609 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" podStartSLOduration=1.897945555 podStartE2EDuration="4.571587248s" totalImagesPullingTime="2.673641693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:58:33.571358622 +0000 UTC m=+369.262049673" watchObservedRunningTime="2026-08-04 17:58:33.571587248 +0000 UTC m=+369.262278301" Aug 04 17:58:34.459613 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:34.459583 2572 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-k5r8f container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 17:58:34.460019 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:34.459635 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" podUID="e191e8f4-ac16-4f98-8c33-0b9a6b313992" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 17:58:34.505890 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:34.505860 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:34.824836 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:34.824785 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:34.973157 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:34.973128 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:35.463800 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:35.463760 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-55bf5c9f58-bfs22" event={"ID":"bb7ec098-1dd9-4212-bbf2-d69562f92b97","Type":"ContainerStarted","Data":"45b8680304ece1243ae512094357fb8b4ec3f42fddfc7abd35fee3c60cf8a393"} Aug 04 17:58:35.464164 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:35.463933 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-55bf5c9f58-bfs22" Aug 04 17:58:35.516730 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:35.516677 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-55bf5c9f58-bfs22" podStartSLOduration=1.4881139 podStartE2EDuration="4.516647705s" totalImagesPullingTime="3.028533805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:58:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:58:35.515304176 +0000 UTC m=+371.205995227" watchObservedRunningTime="2026-08-04 17:58:35.516647705 +0000 UTC m=+371.207338756" Aug 04 17:58:36.333794 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:36.333758 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:36.433323 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:36.433295 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:36.555320 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:36.555287 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:36.747568 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:36.747479 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:36.991462 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:36.991428 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:37.143315 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:37.143283 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:37.459319 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:37.459238 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-k5r8f" Aug 04 17:58:38.222468 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:38.222437 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:38.422323 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:38.422290 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:38.538569 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:38.538488 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:38.717817 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:38.717790 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:38.832830 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:38.832800 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:38.969513 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:38.969486 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:58:46.470308 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:58:46.470278 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-55bf5c9f58-bfs22" Aug 04 17:59:03.905410 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:03.905376 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:03.959736 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:03.959705 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:05.755270 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:05.755236 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:22.797205 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:22.797173 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:22.897069 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:22.897037 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:34.520537 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:34.520500 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:35.318340 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:35.318305 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:38.314580 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:38.314551 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:38.556926 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:38.556882 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:38.656914 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:38.656838 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:38.751859 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:38.751832 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:39.202984 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:39.202951 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:39.331486 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:39.331450 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:40.976306 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:40.976269 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:41.088073 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:41.088044 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:42.572488 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:42.572453 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:42.766907 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:42.766877 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:42.974872 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:42.974782 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:44.922206 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:44.922125 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:46.786194 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:46.786158 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:47.761708 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:47.761645 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:47.856978 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:47.856952 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:48.133377 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:48.133339 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:48.234943 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:48.234915 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:49.482301 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:49.482273 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:49.621638 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:49.621606 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:52.539777 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:52.539746 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:52.640621 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:52.640586 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:52.934647 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:52.934611 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:53.036558 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:53.036531 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:53.172947 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:53.172915 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:53.265132 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:53.265055 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:58.873631 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:58.873602 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 17:59:58.973583 ip-10-0-137-173 kubenswrapper[2572]: I0804 17:59:58.973547 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:03.878930 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:03.878899 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:04.413781 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:04.413747 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:07.350123 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:07.350089 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:07.827763 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:07.827732 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:07.947612 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:07.947581 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:12.084931 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:12.084899 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:12.186318 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:12.186289 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:17.395407 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:17.395369 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:17.843641 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:17.843613 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:17.948909 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:17.948876 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:22.594192 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:22.594156 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:22.694132 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:22.694100 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:26.403840 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:26.403807 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:26.463791 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:26.463762 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:26.992548 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:26.992514 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:27.058371 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:27.058336 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:31.368476 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:31.368445 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:31.619570 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:31.619487 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:31.715947 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:31.715911 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:32.015047 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:32.014963 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:32.112152 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:32.112112 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:32.681684 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:32.681642 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:32.745167 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:32.745136 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:33.107403 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:33.107372 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:33.228440 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:33.228407 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:33.298215 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:33.298182 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:34.019573 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:34.019537 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:34.123898 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:34.123863 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:38.702766 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:38.702733 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:38.757603 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:38.757577 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:42.502908 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:42.502874 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:42.744515 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:42.744477 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:49.176109 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:49.176043 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:49.278215 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:49.278184 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:00:49.797644 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:00:49.797607 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:01:02.054816 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:01:02.054772 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:01:02.362690 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:01:02.362593 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:01:09.477608 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:01:09.477572 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:01:10.414040 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:01:10.413999 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:01:10.476848 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:01:10.476817 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:01:13.590263 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:01:13.590232 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:01:13.671158 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:01:13.671128 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:01:45.484130 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:01:45.484090 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:01:45.587218 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:01:45.587174 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:01:46.858688 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:01:46.858640 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:01:47.246759 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:01:47.246656 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:01:47.744266 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:01:47.744226 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:01:47.839056 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:01:47.839023 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:01:58.976528 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:01:58.976492 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:01:59.144571 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:01:59.144541 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:01:59.247542 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:01:59.247462 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:02:17.777068 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:02:17.777028 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:02:23.015513 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:02:23.015477 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:02:25.850100 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:02:25.850058 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:02:31.704383 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:02:31.704348 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:02:31.802237 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:02:31.802205 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:02:31.925811 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:02:31.925782 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:02:44.482485 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:02:44.482451 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:02:48.862861 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:02:48.862788 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:03:00.425220 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:00.425182 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:03:01.062060 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:01.062026 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:03:04.293066 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:04.293026 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:03:11.235400 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:11.235367 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:03:13.681514 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:13.681472 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:03:13.789904 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:13.789872 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:03:24.195785 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.195755 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds"] Aug 04 18:03:24.199013 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.198994 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" Aug 04 18:03:24.202846 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.202822 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 04 18:03:24.202973 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.202904 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dk8xl\"" Aug 04 18:03:24.202973 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.202911 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Aug 04 18:03:24.203191 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.203171 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 04 18:03:24.210982 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.210959 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds"] Aug 04 18:03:24.232863 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.232833 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:03:24.304287 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.304253 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c7cd685-f67d-4c9f-89bf-4cda37dda6f6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sqcds\" (UID: \"2c7cd685-f67d-4c9f-89bf-4cda37dda6f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" Aug 04 18:03:24.304287 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.304288 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c7cd685-f67d-4c9f-89bf-4cda37dda6f6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sqcds\" (UID: \"2c7cd685-f67d-4c9f-89bf-4cda37dda6f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" Aug 04 18:03:24.304513 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.304332 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smr5m\" (UniqueName: \"kubernetes.io/projected/2c7cd685-f67d-4c9f-89bf-4cda37dda6f6-kube-api-access-smr5m\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sqcds\" (UID: \"2c7cd685-f67d-4c9f-89bf-4cda37dda6f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" Aug 04 18:03:24.304513 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.304411 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c7cd685-f67d-4c9f-89bf-4cda37dda6f6-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sqcds\" (UID: \"2c7cd685-f67d-4c9f-89bf-4cda37dda6f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" Aug 04 18:03:24.304513 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.304459 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c7cd685-f67d-4c9f-89bf-4cda37dda6f6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sqcds\" (UID: \"2c7cd685-f67d-4c9f-89bf-4cda37dda6f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" Aug 04 18:03:24.304627 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.304514 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c7cd685-f67d-4c9f-89bf-4cda37dda6f6-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sqcds\" (UID: \"2c7cd685-f67d-4c9f-89bf-4cda37dda6f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" Aug 04 18:03:24.405631 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.405598 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c7cd685-f67d-4c9f-89bf-4cda37dda6f6-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sqcds\" (UID: \"2c7cd685-f67d-4c9f-89bf-4cda37dda6f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" Aug 04 18:03:24.405842 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.405644 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c7cd685-f67d-4c9f-89bf-4cda37dda6f6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sqcds\" (UID: \"2c7cd685-f67d-4c9f-89bf-4cda37dda6f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" Aug 04 18:03:24.405842 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.405703 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c7cd685-f67d-4c9f-89bf-4cda37dda6f6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sqcds\" (UID: \"2c7cd685-f67d-4c9f-89bf-4cda37dda6f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" Aug 04 18:03:24.405842 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.405779 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smr5m\" (UniqueName: \"kubernetes.io/projected/2c7cd685-f67d-4c9f-89bf-4cda37dda6f6-kube-api-access-smr5m\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sqcds\" (UID: \"2c7cd685-f67d-4c9f-89bf-4cda37dda6f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" Aug 04 18:03:24.405842 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.405816 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c7cd685-f67d-4c9f-89bf-4cda37dda6f6-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sqcds\" (UID: \"2c7cd685-f67d-4c9f-89bf-4cda37dda6f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" Aug 04 18:03:24.406023 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.405982 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c7cd685-f67d-4c9f-89bf-4cda37dda6f6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sqcds\" (UID: \"2c7cd685-f67d-4c9f-89bf-4cda37dda6f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" Aug 04 18:03:24.406202 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.406183 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c7cd685-f67d-4c9f-89bf-4cda37dda6f6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sqcds\" (UID: \"2c7cd685-f67d-4c9f-89bf-4cda37dda6f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" Aug 04 18:03:24.406279 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.406217 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c7cd685-f67d-4c9f-89bf-4cda37dda6f6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sqcds\" (UID: \"2c7cd685-f67d-4c9f-89bf-4cda37dda6f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" Aug 04 18:03:24.406279 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.406246 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c7cd685-f67d-4c9f-89bf-4cda37dda6f6-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sqcds\" (UID: \"2c7cd685-f67d-4c9f-89bf-4cda37dda6f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" Aug 04 18:03:24.408103 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.408085 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c7cd685-f67d-4c9f-89bf-4cda37dda6f6-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sqcds\" (UID: \"2c7cd685-f67d-4c9f-89bf-4cda37dda6f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" Aug 04 18:03:24.408395 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.408378 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c7cd685-f67d-4c9f-89bf-4cda37dda6f6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sqcds\" (UID: \"2c7cd685-f67d-4c9f-89bf-4cda37dda6f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" Aug 04 18:03:24.416311 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.416290 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smr5m\" (UniqueName: \"kubernetes.io/projected/2c7cd685-f67d-4c9f-89bf-4cda37dda6f6-kube-api-access-smr5m\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-sqcds\" (UID: \"2c7cd685-f67d-4c9f-89bf-4cda37dda6f6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" Aug 04 18:03:24.508460 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.508369 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" Aug 04 18:03:24.644516 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.644398 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds"] Aug 04 18:03:24.647031 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:03:24.647007 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c7cd685_f67d_4c9f_89bf_4cda37dda6f6.slice/crio-069db82321ae009d6f89e1395cc2bb396b65dc900cabcc501e43d6c51e989e87 WatchSource:0}: Error finding container 069db82321ae009d6f89e1395cc2bb396b65dc900cabcc501e43d6c51e989e87: Status 404 returned error can't find the container with id 069db82321ae009d6f89e1395cc2bb396b65dc900cabcc501e43d6c51e989e87 Aug 04 18:03:24.648272 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.648257 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 18:03:24.685801 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:24.685777 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" event={"ID":"2c7cd685-f67d-4c9f-89bf-4cda37dda6f6","Type":"ContainerStarted","Data":"069db82321ae009d6f89e1395cc2bb396b65dc900cabcc501e43d6c51e989e87"} Aug 04 18:03:25.691216 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:25.691191 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" event={"ID":"2c7cd685-f67d-4c9f-89bf-4cda37dda6f6","Type":"ContainerStarted","Data":"221a6be8a7231398e980453ed1eff45010fed7e76ab5663c626a9d4400b88fdf"} Aug 04 18:03:25.731196 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:25.731142 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" podStartSLOduration=0.754194706 podStartE2EDuration="1.73112485s" totalImagesPullingTime="976.930144ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:03:25.730238693 +0000 UTC m=+661.420929743" watchObservedRunningTime="2026-08-04 18:03:25.73112485 +0000 UTC m=+661.421815900" Aug 04 18:03:34.508904 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:34.508866 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" Aug 04 18:03:34.509283 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:34.508927 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" Aug 04 18:03:34.521503 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:34.521468 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" Aug 04 18:03:34.742874 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:03:34.742847 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-sqcds" Aug 04 18:04:16.343705 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:04:16.343598 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:04:16.438824 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:04:16.438788 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:04:17.695834 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:04:17.695801 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:04:17.836944 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:04:17.836907 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:04:34.851706 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:04:34.851651 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:04:55.970408 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:04:55.970374 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:06:23.778410 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:06:23.778374 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:06:24.986910 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:06:24.986881 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:07:27.329371 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:07:27.329328 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:07:28.807596 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:07:28.807563 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:07:34.008238 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:07:34.008195 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:07:34.072727 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:07:34.072695 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:07:34.158151 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:07:34.158117 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:07:34.260369 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:07:34.260281 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:07:34.414479 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:07:34.414445 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:07:41.217917 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:07:41.217880 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:07:41.293759 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:07:41.293729 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:09:52.107502 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:09:52.107467 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:44.082134 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:12:44.082092 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:12:45.344167 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:12:45.344131 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:13:03.799751 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:13:03.799711 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:14:55.042317 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:14:55.042220 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:14:55.124542 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:14:55.124510 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:15:00.186113 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:15:00.186078 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764455-45sjp"] Aug 04 18:15:00.189535 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:15:00.189515 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764455-45sjp" Aug 04 18:15:00.208538 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:15:00.208517 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pnbq7\"" Aug 04 18:15:00.208616 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:15:00.208570 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764455-45sjp" Aug 04 18:15:00.272625 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:15:00.272592 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764455-45sjp"] Aug 04 18:15:00.542461 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:15:00.542433 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764455-45sjp"] Aug 04 18:15:00.544959 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:15:00.544929 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2263abad_9b33_4666_8e3e_22b76fe5bae7.slice/crio-e9d9ebfb006c72b24f77059e6f786e6af19480115d4eda2dbc063fd7fab36385 WatchSource:0}: Error finding container e9d9ebfb006c72b24f77059e6f786e6af19480115d4eda2dbc063fd7fab36385: Status 404 returned error can't find the container with id e9d9ebfb006c72b24f77059e6f786e6af19480115d4eda2dbc063fd7fab36385 Aug 04 18:15:00.546849 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:15:00.546831 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 18:15:00.649436 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:15:00.649408 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764455-45sjp" event={"ID":"2263abad-9b33-4666-8e3e-22b76fe5bae7","Type":"ContainerStarted","Data":"e9d9ebfb006c72b24f77059e6f786e6af19480115d4eda2dbc063fd7fab36385"} Aug 04 18:15:11.701376 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:15:11.701332 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764455-45sjp" event={"ID":"2263abad-9b33-4666-8e3e-22b76fe5bae7","Type":"ContainerStarted","Data":"8dc05cc0345a94dffd93da7522a52d230b329ac551166e414ff77d4efb43b870"} Aug 04 18:15:11.721194 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:15:11.721145 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764455-45sjp" podStartSLOduration=1.023841348 podStartE2EDuration="11.721129948s" totalImagesPullingTime="10.6972886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:15:11.720140206 +0000 UTC m=+1367.410831257" watchObservedRunningTime="2026-08-04 18:15:11.721129948 +0000 UTC m=+1367.411820999" Aug 04 18:15:12.706822 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:15:12.706781 2572 generic.go:356] "Generic (PLEG): container finished" podID="2263abad-9b33-4666-8e3e-22b76fe5bae7" containerID="8dc05cc0345a94dffd93da7522a52d230b329ac551166e414ff77d4efb43b870" exitCode=0 Aug 04 18:15:12.707195 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:15:12.706866 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764455-45sjp" event={"ID":"2263abad-9b33-4666-8e3e-22b76fe5bae7","Type":"ContainerDied","Data":"8dc05cc0345a94dffd93da7522a52d230b329ac551166e414ff77d4efb43b870"} Aug 04 18:15:13.838501 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:15:13.838471 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764455-45sjp" Aug 04 18:15:14.718637 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:15:14.718607 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764455-45sjp" Aug 04 18:15:14.718637 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:15:14.718629 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764455-45sjp" event={"ID":"2263abad-9b33-4666-8e3e-22b76fe5bae7","Type":"ContainerDied","Data":"e9d9ebfb006c72b24f77059e6f786e6af19480115d4eda2dbc063fd7fab36385"} Aug 04 18:15:14.718907 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:15:14.718729 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9d9ebfb006c72b24f77059e6f786e6af19480115d4eda2dbc063fd7fab36385" Aug 04 18:17:16.309955 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.309924 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b"] Aug 04 18:17:16.310413 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.310248 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2263abad-9b33-4666-8e3e-22b76fe5bae7" containerName="cleanup" Aug 04 18:17:16.310413 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.310260 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2263abad-9b33-4666-8e3e-22b76fe5bae7" containerName="cleanup" Aug 04 18:17:16.310413 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.310333 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2263abad-9b33-4666-8e3e-22b76fe5bae7" containerName="cleanup" Aug 04 18:17:16.312271 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.312254 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:17:16.314790 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.314772 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-b751fcdc-kserve-self-signed-certs\"" Aug 04 18:17:16.336857 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.336836 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b"] Aug 04 18:17:16.423223 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.423188 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0606fd5a-98d5-4823-87d1-8455601c7315-dshm\") pod \"e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b\" (UID: \"0606fd5a-98d5-4823-87d1-8455601c7315\") " pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:17:16.423223 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.423227 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0606fd5a-98d5-4823-87d1-8455601c7315-model-cache\") pod \"e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b\" (UID: \"0606fd5a-98d5-4823-87d1-8455601c7315\") " pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:17:16.423429 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.423264 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0606fd5a-98d5-4823-87d1-8455601c7315-home\") pod \"e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b\" (UID: \"0606fd5a-98d5-4823-87d1-8455601c7315\") " pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:17:16.423429 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.423287 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0606fd5a-98d5-4823-87d1-8455601c7315-tmp-dir\") pod \"e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b\" (UID: \"0606fd5a-98d5-4823-87d1-8455601c7315\") " pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:17:16.423429 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.423332 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs7ch\" (UniqueName: \"kubernetes.io/projected/0606fd5a-98d5-4823-87d1-8455601c7315-kube-api-access-vs7ch\") pod \"e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b\" (UID: \"0606fd5a-98d5-4823-87d1-8455601c7315\") " pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:17:16.423429 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.423357 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0606fd5a-98d5-4823-87d1-8455601c7315-tls-certs\") pod \"e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b\" (UID: \"0606fd5a-98d5-4823-87d1-8455601c7315\") " pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:17:16.523984 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.523956 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0606fd5a-98d5-4823-87d1-8455601c7315-dshm\") pod \"e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b\" (UID: \"0606fd5a-98d5-4823-87d1-8455601c7315\") " pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:17:16.524174 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.523990 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0606fd5a-98d5-4823-87d1-8455601c7315-model-cache\") pod \"e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b\" (UID: \"0606fd5a-98d5-4823-87d1-8455601c7315\") " pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:17:16.524174 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.524017 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0606fd5a-98d5-4823-87d1-8455601c7315-home\") pod \"e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b\" (UID: \"0606fd5a-98d5-4823-87d1-8455601c7315\") " pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:17:16.524174 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.524034 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0606fd5a-98d5-4823-87d1-8455601c7315-tmp-dir\") pod \"e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b\" (UID: \"0606fd5a-98d5-4823-87d1-8455601c7315\") " pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:17:16.524174 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.524146 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs7ch\" (UniqueName: \"kubernetes.io/projected/0606fd5a-98d5-4823-87d1-8455601c7315-kube-api-access-vs7ch\") pod \"e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b\" (UID: \"0606fd5a-98d5-4823-87d1-8455601c7315\") " pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:17:16.524366 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.524176 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0606fd5a-98d5-4823-87d1-8455601c7315-tls-certs\") pod \"e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b\" (UID: \"0606fd5a-98d5-4823-87d1-8455601c7315\") " pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:17:16.524453 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.524430 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0606fd5a-98d5-4823-87d1-8455601c7315-model-cache\") pod \"e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b\" (UID: \"0606fd5a-98d5-4823-87d1-8455601c7315\") " pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:17:16.524539 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.524516 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0606fd5a-98d5-4823-87d1-8455601c7315-tmp-dir\") pod \"e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b\" (UID: \"0606fd5a-98d5-4823-87d1-8455601c7315\") " pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:17:16.524598 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.524541 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0606fd5a-98d5-4823-87d1-8455601c7315-home\") pod \"e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b\" (UID: \"0606fd5a-98d5-4823-87d1-8455601c7315\") " pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:17:16.526484 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.526461 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0606fd5a-98d5-4823-87d1-8455601c7315-dshm\") pod \"e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b\" (UID: \"0606fd5a-98d5-4823-87d1-8455601c7315\") " pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:17:16.526754 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.526738 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0606fd5a-98d5-4823-87d1-8455601c7315-tls-certs\") pod \"e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b\" (UID: \"0606fd5a-98d5-4823-87d1-8455601c7315\") " pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:17:16.535127 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.535109 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs7ch\" (UniqueName: \"kubernetes.io/projected/0606fd5a-98d5-4823-87d1-8455601c7315-kube-api-access-vs7ch\") pod \"e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b\" (UID: \"0606fd5a-98d5-4823-87d1-8455601c7315\") " pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:17:16.621124 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.621043 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:17:16.751257 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:16.751230 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b"] Aug 04 18:17:16.753327 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:17:16.753299 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0606fd5a_98d5_4823_87d1_8455601c7315.slice/crio-84fdb45800cd1b3f11d8ec668eac8ae2a409952a56308ec3d815b64d91a45eb4 WatchSource:0}: Error finding container 84fdb45800cd1b3f11d8ec668eac8ae2a409952a56308ec3d815b64d91a45eb4: Status 404 returned error can't find the container with id 84fdb45800cd1b3f11d8ec668eac8ae2a409952a56308ec3d815b64d91a45eb4 Aug 04 18:17:17.262457 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:17.262425 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" event={"ID":"0606fd5a-98d5-4823-87d1-8455601c7315","Type":"ContainerStarted","Data":"0e96812ed08a51fb0ee45d1d4a5253d9d0040f418c5396dce8c1a7a3e1033edf"} Aug 04 18:17:17.262457 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:17.262460 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" event={"ID":"0606fd5a-98d5-4823-87d1-8455601c7315","Type":"ContainerStarted","Data":"84fdb45800cd1b3f11d8ec668eac8ae2a409952a56308ec3d815b64d91a45eb4"} Aug 04 18:17:17.289213 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:17.289157 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" podStartSLOduration=1.289141782 podStartE2EDuration="1.289141782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:17:17.288444724 +0000 UTC m=+1492.979135775" watchObservedRunningTime="2026-08-04 18:17:17.289141782 +0000 UTC m=+1492.979832832" Aug 04 18:17:17.379945 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:17.379910 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:25.335110 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:25.335076 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:26.621453 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:26.621377 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:17:26.621453 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:26.621424 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:17:26.633871 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:26.633849 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:17:27.317750 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:27.317727 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:17:46.967185 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:46.967113 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:58.562910 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:58.562881 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b"] Aug 04 18:17:58.583390 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:58.583362 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:17:59.158106 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:17:59.158073 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:13.582160 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:13.582102 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" podUID="0606fd5a-98d5-4823-87d1-8455601c7315" containerName="main" containerID="cri-o://0e96812ed08a51fb0ee45d1d4a5253d9d0040f418c5396dce8c1a7a3e1033edf" gracePeriod=45 Aug 04 18:18:13.817333 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:13.817301 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:18:13.933245 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:13.933158 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vs7ch\" (UniqueName: \"kubernetes.io/projected/0606fd5a-98d5-4823-87d1-8455601c7315-kube-api-access-vs7ch\") pod \"0606fd5a-98d5-4823-87d1-8455601c7315\" (UID: \"0606fd5a-98d5-4823-87d1-8455601c7315\") " Aug 04 18:18:13.933245 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:13.933219 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0606fd5a-98d5-4823-87d1-8455601c7315-model-cache\") pod \"0606fd5a-98d5-4823-87d1-8455601c7315\" (UID: \"0606fd5a-98d5-4823-87d1-8455601c7315\") " Aug 04 18:18:13.933454 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:13.933258 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0606fd5a-98d5-4823-87d1-8455601c7315-home\") pod \"0606fd5a-98d5-4823-87d1-8455601c7315\" (UID: \"0606fd5a-98d5-4823-87d1-8455601c7315\") " Aug 04 18:18:13.933454 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:13.933280 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0606fd5a-98d5-4823-87d1-8455601c7315-dshm\") pod \"0606fd5a-98d5-4823-87d1-8455601c7315\" (UID: \"0606fd5a-98d5-4823-87d1-8455601c7315\") " Aug 04 18:18:13.933454 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:13.933342 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0606fd5a-98d5-4823-87d1-8455601c7315-tmp-dir\") pod \"0606fd5a-98d5-4823-87d1-8455601c7315\" (UID: \"0606fd5a-98d5-4823-87d1-8455601c7315\") " Aug 04 18:18:13.933454 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:13.933373 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0606fd5a-98d5-4823-87d1-8455601c7315-tls-certs\") pod \"0606fd5a-98d5-4823-87d1-8455601c7315\" (UID: \"0606fd5a-98d5-4823-87d1-8455601c7315\") " Aug 04 18:18:13.933647 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:13.933519 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0606fd5a-98d5-4823-87d1-8455601c7315-home" (OuterVolumeSpecName: "home") pod "0606fd5a-98d5-4823-87d1-8455601c7315" (UID: "0606fd5a-98d5-4823-87d1-8455601c7315"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:18:13.933647 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:13.933564 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0606fd5a-98d5-4823-87d1-8455601c7315-model-cache" (OuterVolumeSpecName: "model-cache") pod "0606fd5a-98d5-4823-87d1-8455601c7315" (UID: "0606fd5a-98d5-4823-87d1-8455601c7315"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:18:13.933647 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:13.933634 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0606fd5a-98d5-4823-87d1-8455601c7315-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0606fd5a-98d5-4823-87d1-8455601c7315" (UID: "0606fd5a-98d5-4823-87d1-8455601c7315"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:18:13.933873 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:13.933851 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0606fd5a-98d5-4823-87d1-8455601c7315-model-cache\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:18:13.933933 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:13.933877 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0606fd5a-98d5-4823-87d1-8455601c7315-home\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:18:13.933933 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:13.933893 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0606fd5a-98d5-4823-87d1-8455601c7315-tmp-dir\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:18:13.935682 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:13.935642 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0606fd5a-98d5-4823-87d1-8455601c7315-dshm" (OuterVolumeSpecName: "dshm") pod "0606fd5a-98d5-4823-87d1-8455601c7315" (UID: "0606fd5a-98d5-4823-87d1-8455601c7315"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:18:13.936040 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:13.936012 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0606fd5a-98d5-4823-87d1-8455601c7315-kube-api-access-vs7ch" (OuterVolumeSpecName: "kube-api-access-vs7ch") pod "0606fd5a-98d5-4823-87d1-8455601c7315" (UID: "0606fd5a-98d5-4823-87d1-8455601c7315"). InnerVolumeSpecName "kube-api-access-vs7ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:18:13.936089 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:13.936016 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0606fd5a-98d5-4823-87d1-8455601c7315-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0606fd5a-98d5-4823-87d1-8455601c7315" (UID: "0606fd5a-98d5-4823-87d1-8455601c7315"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:18:14.034643 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:14.034612 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs7ch\" (UniqueName: \"kubernetes.io/projected/0606fd5a-98d5-4823-87d1-8455601c7315-kube-api-access-vs7ch\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:18:14.034643 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:14.034643 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0606fd5a-98d5-4823-87d1-8455601c7315-dshm\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:18:14.034808 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:14.034658 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0606fd5a-98d5-4823-87d1-8455601c7315-tls-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:18:14.513936 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:14.513902 2572 generic.go:356] "Generic (PLEG): container finished" podID="0606fd5a-98d5-4823-87d1-8455601c7315" containerID="0e96812ed08a51fb0ee45d1d4a5253d9d0040f418c5396dce8c1a7a3e1033edf" exitCode=0 Aug 04 18:18:14.514161 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:14.513957 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" Aug 04 18:18:14.514161 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:14.513992 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" event={"ID":"0606fd5a-98d5-4823-87d1-8455601c7315","Type":"ContainerDied","Data":"0e96812ed08a51fb0ee45d1d4a5253d9d0040f418c5396dce8c1a7a3e1033edf"} Aug 04 18:18:14.514161 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:14.514038 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b" event={"ID":"0606fd5a-98d5-4823-87d1-8455601c7315","Type":"ContainerDied","Data":"84fdb45800cd1b3f11d8ec668eac8ae2a409952a56308ec3d815b64d91a45eb4"} Aug 04 18:18:14.514161 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:14.514056 2572 scope.go:117] "RemoveContainer" containerID="0e96812ed08a51fb0ee45d1d4a5253d9d0040f418c5396dce8c1a7a3e1033edf" Aug 04 18:18:14.523499 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:14.523483 2572 scope.go:117] "RemoveContainer" containerID="0e96812ed08a51fb0ee45d1d4a5253d9d0040f418c5396dce8c1a7a3e1033edf" Aug 04 18:18:14.523945 ip-10-0-137-173 kubenswrapper[2572]: E0804 18:18:14.523922 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e96812ed08a51fb0ee45d1d4a5253d9d0040f418c5396dce8c1a7a3e1033edf\": container with ID starting with 0e96812ed08a51fb0ee45d1d4a5253d9d0040f418c5396dce8c1a7a3e1033edf not found: ID does not exist" containerID="0e96812ed08a51fb0ee45d1d4a5253d9d0040f418c5396dce8c1a7a3e1033edf" Aug 04 18:18:14.524020 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:14.523952 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e96812ed08a51fb0ee45d1d4a5253d9d0040f418c5396dce8c1a7a3e1033edf"} err="failed to get container status \"0e96812ed08a51fb0ee45d1d4a5253d9d0040f418c5396dce8c1a7a3e1033edf\": rpc error: code = NotFound desc = could not find container \"0e96812ed08a51fb0ee45d1d4a5253d9d0040f418c5396dce8c1a7a3e1033edf\": container with ID starting with 0e96812ed08a51fb0ee45d1d4a5253d9d0040f418c5396dce8c1a7a3e1033edf not found: ID does not exist" Aug 04 18:18:14.536470 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:14.536447 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b"] Aug 04 18:18:14.543362 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:14.543341 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-b751fcdc-kserve-7cf89b4d75-66b7b"] Aug 04 18:18:14.822324 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:14.822292 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0606fd5a-98d5-4823-87d1-8455601c7315" path="/var/lib/kubelet/pods/0606fd5a-98d5-4823-87d1-8455601c7315/volumes" Aug 04 18:18:35.493062 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:35.493032 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:18:35.591653 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:18:35.591625 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:21:26.055329 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:21:26.055300 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:21:51.988891 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:21:51.988858 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:21:55.094728 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:21:55.094693 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:21:55.188355 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:21:55.188304 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:21:55.536731 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:21:55.536630 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:02.538040 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:02.538004 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:02.637936 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:02.637900 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:02.774467 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:02.774435 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:03.090855 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:03.090817 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:03.454504 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:03.454425 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:05.517284 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:05.517254 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:05.614658 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:05.614624 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:06.655106 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:06.655067 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:06.936936 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:06.936852 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:07.180987 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:07.180956 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:07.280488 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:07.280401 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:30.968395 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:30.968361 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745b1b91-tjbkm"] Aug 04 18:22:30.968880 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:30.968856 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0606fd5a-98d5-4823-87d1-8455601c7315" containerName="main" Aug 04 18:22:30.968880 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:30.968874 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0606fd5a-98d5-4823-87d1-8455601c7315" containerName="main" Aug 04 18:22:30.968991 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:30.968951 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0606fd5a-98d5-4823-87d1-8455601c7315" containerName="main" Aug 04 18:22:30.970851 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:30.970830 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745b1b91-tjbkm" Aug 04 18:22:30.973691 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:30.973657 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pnbq7\"" Aug 04 18:22:30.973911 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:30.973897 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 18:22:30.979318 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:30.979285 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745b1b91-tjbkm"] Aug 04 18:22:31.051364 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:31.051336 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/738670be-656b-42b2-a614-49b82a1b34a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-745b1b91-tjbkm\" (UID: \"738670be-656b-42b2-a614-49b82a1b34a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745b1b91-tjbkm" Aug 04 18:22:31.152271 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:31.152244 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/738670be-656b-42b2-a614-49b82a1b34a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-745b1b91-tjbkm\" (UID: \"738670be-656b-42b2-a614-49b82a1b34a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745b1b91-tjbkm" Aug 04 18:22:31.152915 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:31.152896 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/738670be-656b-42b2-a614-49b82a1b34a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-745b1b91-tjbkm\" (UID: \"738670be-656b-42b2-a614-49b82a1b34a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745b1b91-tjbkm" Aug 04 18:22:31.282472 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:31.282400 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745b1b91-tjbkm" Aug 04 18:22:31.408966 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:31.408932 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745b1b91-tjbkm"] Aug 04 18:22:31.412030 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:22:31.412006 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod738670be_656b_42b2_a614_49b82a1b34a9.slice/crio-58f15c5969f0d87597b8400730211750f40cc4e19e24aea69131a3958391994a WatchSource:0}: Error finding container 58f15c5969f0d87597b8400730211750f40cc4e19e24aea69131a3958391994a: Status 404 returned error can't find the container with id 58f15c5969f0d87597b8400730211750f40cc4e19e24aea69131a3958391994a Aug 04 18:22:31.638334 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:31.638309 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745b1b91-tjbkm" event={"ID":"738670be-656b-42b2-a614-49b82a1b34a9","Type":"ContainerStarted","Data":"e4be6246c56705ac73a5bcc4e8f3522cae5708b03c006d8b05fd754ec3b97b30"} Aug 04 18:22:31.638428 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:31.638367 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745b1b91-tjbkm" event={"ID":"738670be-656b-42b2-a614-49b82a1b34a9","Type":"ContainerStarted","Data":"58f15c5969f0d87597b8400730211750f40cc4e19e24aea69131a3958391994a"} Aug 04 18:22:32.644362 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:32.644325 2572 generic.go:356] "Generic (PLEG): container finished" podID="738670be-656b-42b2-a614-49b82a1b34a9" containerID="e4be6246c56705ac73a5bcc4e8f3522cae5708b03c006d8b05fd754ec3b97b30" exitCode=0 Aug 04 18:22:32.644797 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:32.644413 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745b1b91-tjbkm" event={"ID":"738670be-656b-42b2-a614-49b82a1b34a9","Type":"ContainerDied","Data":"e4be6246c56705ac73a5bcc4e8f3522cae5708b03c006d8b05fd754ec3b97b30"} Aug 04 18:22:33.778451 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:33.778427 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745b1b91-tjbkm" Aug 04 18:22:33.876418 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:33.876387 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/738670be-656b-42b2-a614-49b82a1b34a9-maas-api-service-ca\") pod \"738670be-656b-42b2-a614-49b82a1b34a9\" (UID: \"738670be-656b-42b2-a614-49b82a1b34a9\") " Aug 04 18:22:33.876758 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:33.876734 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/738670be-656b-42b2-a614-49b82a1b34a9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "738670be-656b-42b2-a614-49b82a1b34a9" (UID: "738670be-656b-42b2-a614-49b82a1b34a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:22:33.977829 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:33.977776 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/738670be-656b-42b2-a614-49b82a1b34a9-maas-api-service-ca\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:22:34.654074 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:34.654044 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745b1b91-tjbkm" Aug 04 18:22:34.654074 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:34.654066 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745b1b91-tjbkm" event={"ID":"738670be-656b-42b2-a614-49b82a1b34a9","Type":"ContainerDied","Data":"58f15c5969f0d87597b8400730211750f40cc4e19e24aea69131a3958391994a"} Aug 04 18:22:34.654271 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:34.654109 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58f15c5969f0d87597b8400730211750f40cc4e19e24aea69131a3958391994a" Aug 04 18:22:37.408398 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:37.408368 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:37.488482 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:37.488439 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:37.589354 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:37.589316 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:37.640743 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:37.640711 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745b1b91-tjbkm"] Aug 04 18:22:37.643449 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:37.643427 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-745b1b91-tjbkm"] Aug 04 18:22:37.651555 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:37.651535 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:37.794549 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:37.794521 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:38.020924 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:38.020889 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:38.251406 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:38.251314 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:38.359975 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:38.359946 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:38.496998 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:38.496965 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:38.804950 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:38.804915 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:38.821277 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:38.821253 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="738670be-656b-42b2-a614-49b82a1b34a9" path="/var/lib/kubelet/pods/738670be-656b-42b2-a614-49b82a1b34a9/volumes" Aug 04 18:22:38.902535 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:38.902502 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:39.111726 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:39.111633 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:41.740851 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:41.740815 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:41.892857 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:41.892828 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:42.024308 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:42.024229 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:42.433201 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:42.433163 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:42.527343 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:42.527311 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:22:54.176607 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:22:54.176566 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:06.732996 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:06.732964 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a9c503-qdmtl"] Aug 04 18:23:06.733491 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:06.733322 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="738670be-656b-42b2-a614-49b82a1b34a9" containerName="revoke-keys" Aug 04 18:23:06.733491 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:06.733333 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="738670be-656b-42b2-a614-49b82a1b34a9" containerName="revoke-keys" Aug 04 18:23:06.733491 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:06.733389 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="738670be-656b-42b2-a614-49b82a1b34a9" containerName="revoke-keys" Aug 04 18:23:06.736192 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:06.736177 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a9c503-qdmtl" Aug 04 18:23:06.738984 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:06.738961 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 18:23:06.739540 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:06.739525 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pnbq7\"" Aug 04 18:23:06.746603 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:06.746582 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a9c503-qdmtl"] Aug 04 18:23:06.757988 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:06.757961 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e539119-7880-4cbc-b205-73f78d50f775-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-16a9c503-qdmtl\" (UID: \"0e539119-7880-4cbc-b205-73f78d50f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a9c503-qdmtl" Aug 04 18:23:06.858948 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:06.858916 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e539119-7880-4cbc-b205-73f78d50f775-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-16a9c503-qdmtl\" (UID: \"0e539119-7880-4cbc-b205-73f78d50f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a9c503-qdmtl" Aug 04 18:23:06.859474 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:06.859457 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e539119-7880-4cbc-b205-73f78d50f775-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-16a9c503-qdmtl\" (UID: \"0e539119-7880-4cbc-b205-73f78d50f775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a9c503-qdmtl" Aug 04 18:23:07.047286 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:07.047259 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a9c503-qdmtl" Aug 04 18:23:07.176930 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:07.176908 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a9c503-qdmtl"] Aug 04 18:23:07.179029 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:23:07.179003 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e539119_7880_4cbc_b205_73f78d50f775.slice/crio-d09ba91bc915805e827856b3b3ce0849a9e417caea3d1109083102756bd2fa75 WatchSource:0}: Error finding container d09ba91bc915805e827856b3b3ce0849a9e417caea3d1109083102756bd2fa75: Status 404 returned error can't find the container with id d09ba91bc915805e827856b3b3ce0849a9e417caea3d1109083102756bd2fa75 Aug 04 18:23:07.799210 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:07.799181 2572 generic.go:356] "Generic (PLEG): container finished" podID="0e539119-7880-4cbc-b205-73f78d50f775" containerID="d42300ab548bc1627e5bfc9a31de9c892cb2451e2a21c758d8859da00a1f7643" exitCode=0 Aug 04 18:23:07.799506 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:07.799260 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a9c503-qdmtl" event={"ID":"0e539119-7880-4cbc-b205-73f78d50f775","Type":"ContainerDied","Data":"d42300ab548bc1627e5bfc9a31de9c892cb2451e2a21c758d8859da00a1f7643"} Aug 04 18:23:07.799506 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:07.799307 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a9c503-qdmtl" event={"ID":"0e539119-7880-4cbc-b205-73f78d50f775","Type":"ContainerStarted","Data":"d09ba91bc915805e827856b3b3ce0849a9e417caea3d1109083102756bd2fa75"} Aug 04 18:23:08.928844 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:08.928819 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a9c503-qdmtl" Aug 04 18:23:08.976650 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:08.976625 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e539119-7880-4cbc-b205-73f78d50f775-maas-api-service-ca\") pod \"0e539119-7880-4cbc-b205-73f78d50f775\" (UID: \"0e539119-7880-4cbc-b205-73f78d50f775\") " Aug 04 18:23:08.976978 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:08.976953 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e539119-7880-4cbc-b205-73f78d50f775-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e539119-7880-4cbc-b205-73f78d50f775" (UID: "0e539119-7880-4cbc-b205-73f78d50f775"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:23:09.078036 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:09.077977 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e539119-7880-4cbc-b205-73f78d50f775-maas-api-service-ca\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:23:09.809513 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:09.809481 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a9c503-qdmtl" event={"ID":"0e539119-7880-4cbc-b205-73f78d50f775","Type":"ContainerDied","Data":"d09ba91bc915805e827856b3b3ce0849a9e417caea3d1109083102756bd2fa75"} Aug 04 18:23:09.809513 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:09.809511 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a9c503-qdmtl" Aug 04 18:23:09.809734 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:09.809526 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d09ba91bc915805e827856b3b3ce0849a9e417caea3d1109083102756bd2fa75" Aug 04 18:23:12.760986 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:12.760948 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:12.839421 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:12.839389 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:12.899627 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:12.899593 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:12.990502 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:12.990467 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a9c503-qdmtl"] Aug 04 18:23:12.994522 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:12.994495 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-16a9c503-qdmtl"] Aug 04 18:23:12.994722 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:12.994702 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:13.146072 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:13.146039 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:13.366769 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:13.366736 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:13.594266 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:13.594235 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:13.897463 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:13.897436 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:13.999688 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:13.999539 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:14.823017 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:14.822988 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e539119-7880-4cbc-b205-73f78d50f775" path="/var/lib/kubelet/pods/0e539119-7880-4cbc-b205-73f78d50f775/volumes" Aug 04 18:23:15.827833 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:15.827793 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:15.931503 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:15.931475 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:16.991065 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:16.991032 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:17.289733 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:17.289646 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:17.641912 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:17.641868 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:17.741056 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:17.741018 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:41.643505 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:41.643468 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bcb508fd-6dbfn"] Aug 04 18:23:41.644012 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:41.643842 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e539119-7880-4cbc-b205-73f78d50f775" containerName="revoke-keys" Aug 04 18:23:41.644012 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:41.643855 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e539119-7880-4cbc-b205-73f78d50f775" containerName="revoke-keys" Aug 04 18:23:41.644012 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:41.643934 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e539119-7880-4cbc-b205-73f78d50f775" containerName="revoke-keys" Aug 04 18:23:41.646727 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:41.646704 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bcb508fd-6dbfn" Aug 04 18:23:41.650737 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:41.650708 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 18:23:41.650836 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:41.650770 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pnbq7\"" Aug 04 18:23:41.657443 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:41.657422 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bcb508fd-6dbfn"] Aug 04 18:23:41.748187 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:41.748153 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a69dfa0b-1c1c-4af6-b7d1-a68b6880c866-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bcb508fd-6dbfn\" (UID: \"a69dfa0b-1c1c-4af6-b7d1-a68b6880c866\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bcb508fd-6dbfn" Aug 04 18:23:41.848827 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:41.848787 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a69dfa0b-1c1c-4af6-b7d1-a68b6880c866-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bcb508fd-6dbfn\" (UID: \"a69dfa0b-1c1c-4af6-b7d1-a68b6880c866\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bcb508fd-6dbfn" Aug 04 18:23:41.849422 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:41.849403 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a69dfa0b-1c1c-4af6-b7d1-a68b6880c866-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bcb508fd-6dbfn\" (UID: \"a69dfa0b-1c1c-4af6-b7d1-a68b6880c866\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bcb508fd-6dbfn" Aug 04 18:23:41.957691 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:41.957600 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bcb508fd-6dbfn" Aug 04 18:23:42.087689 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:42.087638 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bcb508fd-6dbfn"] Aug 04 18:23:42.089939 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:23:42.089911 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda69dfa0b_1c1c_4af6_b7d1_a68b6880c866.slice/crio-19d110230bb0dafee3b7fbc2e2d90653b4a1380fd85811073a0fe2714217266b WatchSource:0}: Error finding container 19d110230bb0dafee3b7fbc2e2d90653b4a1380fd85811073a0fe2714217266b: Status 404 returned error can't find the container with id 19d110230bb0dafee3b7fbc2e2d90653b4a1380fd85811073a0fe2714217266b Aug 04 18:23:42.952688 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:42.952571 2572 generic.go:356] "Generic (PLEG): container finished" podID="a69dfa0b-1c1c-4af6-b7d1-a68b6880c866" containerID="caead1817dac7ea609972cf58ce19f3464a8847003f359f7cf0e579cab522442" exitCode=0 Aug 04 18:23:42.952688 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:42.952646 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bcb508fd-6dbfn" event={"ID":"a69dfa0b-1c1c-4af6-b7d1-a68b6880c866","Type":"ContainerDied","Data":"caead1817dac7ea609972cf58ce19f3464a8847003f359f7cf0e579cab522442"} Aug 04 18:23:42.953078 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:42.952717 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bcb508fd-6dbfn" event={"ID":"a69dfa0b-1c1c-4af6-b7d1-a68b6880c866","Type":"ContainerStarted","Data":"19d110230bb0dafee3b7fbc2e2d90653b4a1380fd85811073a0fe2714217266b"} Aug 04 18:23:44.089211 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:44.089187 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bcb508fd-6dbfn" Aug 04 18:23:44.174301 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:44.174272 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a69dfa0b-1c1c-4af6-b7d1-a68b6880c866-maas-api-service-ca\") pod \"a69dfa0b-1c1c-4af6-b7d1-a68b6880c866\" (UID: \"a69dfa0b-1c1c-4af6-b7d1-a68b6880c866\") " Aug 04 18:23:44.174584 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:44.174561 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a69dfa0b-1c1c-4af6-b7d1-a68b6880c866-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a69dfa0b-1c1c-4af6-b7d1-a68b6880c866" (UID: "a69dfa0b-1c1c-4af6-b7d1-a68b6880c866"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:23:44.275614 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:44.275543 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a69dfa0b-1c1c-4af6-b7d1-a68b6880c866-maas-api-service-ca\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:23:44.964449 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:44.964417 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bcb508fd-6dbfn" event={"ID":"a69dfa0b-1c1c-4af6-b7d1-a68b6880c866","Type":"ContainerDied","Data":"19d110230bb0dafee3b7fbc2e2d90653b4a1380fd85811073a0fe2714217266b"} Aug 04 18:23:44.964610 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:44.964464 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19d110230bb0dafee3b7fbc2e2d90653b4a1380fd85811073a0fe2714217266b" Aug 04 18:23:44.964610 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:44.964425 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bcb508fd-6dbfn" Aug 04 18:23:47.854427 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:47.854394 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:47.928472 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:47.928441 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:48.000402 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:48.000375 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:48.054247 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:48.054222 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bcb508fd-6dbfn"] Aug 04 18:23:48.059592 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:48.059569 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bcb508fd-6dbfn"] Aug 04 18:23:48.060003 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:48.059971 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:48.823155 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:48.823108 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a69dfa0b-1c1c-4af6-b7d1-a68b6880c866" path="/var/lib/kubelet/pods/a69dfa0b-1c1c-4af6-b7d1-a68b6880c866/volumes" Aug 04 18:23:49.239818 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:49.239726 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:49.450856 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:49.450822 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:49.659412 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:49.659380 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:49.972581 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:49.972491 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:50.230853 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:50.230770 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:52.076768 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:52.076730 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:52.188091 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:52.188063 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:53.231437 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:53.231408 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:53.485013 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:53.484931 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:53.847104 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:53.847072 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:23:53.946055 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:23:53.946023 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:24:16.932641 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:16.932609 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-07becd73-7hfjx"] Aug 04 18:24:16.933061 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:16.932940 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a69dfa0b-1c1c-4af6-b7d1-a68b6880c866" containerName="revoke-keys" Aug 04 18:24:16.933061 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:16.932953 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a69dfa0b-1c1c-4af6-b7d1-a68b6880c866" containerName="revoke-keys" Aug 04 18:24:16.933061 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:16.933022 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a69dfa0b-1c1c-4af6-b7d1-a68b6880c866" containerName="revoke-keys" Aug 04 18:24:16.936000 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:16.935984 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-07becd73-7hfjx" Aug 04 18:24:16.939194 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:16.939177 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pnbq7\"" Aug 04 18:24:16.939368 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:16.939352 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 18:24:16.946033 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:16.946013 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-07becd73-7hfjx"] Aug 04 18:24:17.040861 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:17.040825 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14cab2dc-f0b2-44e7-bbd3-b2d598e465a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-07becd73-7hfjx\" (UID: \"14cab2dc-f0b2-44e7-bbd3-b2d598e465a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-07becd73-7hfjx" Aug 04 18:24:17.141989 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:17.141963 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14cab2dc-f0b2-44e7-bbd3-b2d598e465a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-07becd73-7hfjx\" (UID: \"14cab2dc-f0b2-44e7-bbd3-b2d598e465a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-07becd73-7hfjx" Aug 04 18:24:17.142547 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:17.142531 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14cab2dc-f0b2-44e7-bbd3-b2d598e465a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-07becd73-7hfjx\" (UID: \"14cab2dc-f0b2-44e7-bbd3-b2d598e465a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-07becd73-7hfjx" Aug 04 18:24:17.246961 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:17.246895 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-07becd73-7hfjx" Aug 04 18:24:17.381426 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:17.381401 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-07becd73-7hfjx"] Aug 04 18:24:17.383219 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:24:17.383195 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14cab2dc_f0b2_44e7_bbd3_b2d598e465a5.slice/crio-4e6ec200b0b9530d46fa378b19ea4363fea4a0050699becf4253536e796fcc12 WatchSource:0}: Error finding container 4e6ec200b0b9530d46fa378b19ea4363fea4a0050699becf4253536e796fcc12: Status 404 returned error can't find the container with id 4e6ec200b0b9530d46fa378b19ea4363fea4a0050699becf4253536e796fcc12 Aug 04 18:24:18.108835 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:18.108806 2572 generic.go:356] "Generic (PLEG): container finished" podID="14cab2dc-f0b2-44e7-bbd3-b2d598e465a5" containerID="00478d1c815290d498487d55bdcbe241f10dd4c4d8e5bed70eafefe1f7a19f69" exitCode=0 Aug 04 18:24:18.109200 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:18.108881 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-07becd73-7hfjx" event={"ID":"14cab2dc-f0b2-44e7-bbd3-b2d598e465a5","Type":"ContainerDied","Data":"00478d1c815290d498487d55bdcbe241f10dd4c4d8e5bed70eafefe1f7a19f69"} Aug 04 18:24:18.109200 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:18.108920 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-07becd73-7hfjx" event={"ID":"14cab2dc-f0b2-44e7-bbd3-b2d598e465a5","Type":"ContainerStarted","Data":"4e6ec200b0b9530d46fa378b19ea4363fea4a0050699becf4253536e796fcc12"} Aug 04 18:24:19.245014 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:19.244991 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-07becd73-7hfjx" Aug 04 18:24:19.359988 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:19.359955 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14cab2dc-f0b2-44e7-bbd3-b2d598e465a5-maas-api-service-ca\") pod \"14cab2dc-f0b2-44e7-bbd3-b2d598e465a5\" (UID: \"14cab2dc-f0b2-44e7-bbd3-b2d598e465a5\") " Aug 04 18:24:19.360323 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:19.360300 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14cab2dc-f0b2-44e7-bbd3-b2d598e465a5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "14cab2dc-f0b2-44e7-bbd3-b2d598e465a5" (UID: "14cab2dc-f0b2-44e7-bbd3-b2d598e465a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:24:19.461392 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:19.461321 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14cab2dc-f0b2-44e7-bbd3-b2d598e465a5-maas-api-service-ca\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:24:20.118752 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:20.118717 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-07becd73-7hfjx" event={"ID":"14cab2dc-f0b2-44e7-bbd3-b2d598e465a5","Type":"ContainerDied","Data":"4e6ec200b0b9530d46fa378b19ea4363fea4a0050699becf4253536e796fcc12"} Aug 04 18:24:20.118752 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:20.118770 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e6ec200b0b9530d46fa378b19ea4363fea4a0050699becf4253536e796fcc12" Aug 04 18:24:20.118980 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:20.118767 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-07becd73-7hfjx" Aug 04 18:24:23.983054 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:23.983021 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:24:24.114185 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:24.114156 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:24:24.215344 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:24.215316 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:24:24.265704 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:24.265624 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-07becd73-7hfjx"] Aug 04 18:24:24.268587 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:24.268562 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-07becd73-7hfjx"] Aug 04 18:24:24.276707 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:24.276684 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:24:24.405177 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:24.405147 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:24:24.646218 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:24.646182 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:24:24.826228 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:24.826195 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14cab2dc-f0b2-44e7-bbd3-b2d598e465a5" path="/var/lib/kubelet/pods/14cab2dc-f0b2-44e7-bbd3-b2d598e465a5/volumes" Aug 04 18:24:25.099875 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:25.099829 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:24:25.201186 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:25.201159 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:24:25.297825 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:25.297794 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:24:36.979052 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:36.979014 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:24:44.359179 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:44.359146 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:24:44.465054 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:44.465017 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:24:44.561406 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:44.561377 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:24:44.660520 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:44.660424 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:24:45.044927 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:24:45.044880 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:25:11.044822 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:25:11.044788 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:25:16.654999 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:25:16.654896 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:25:18.544523 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:25:18.544470 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:25:18.646997 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:25:18.646964 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:25:18.742528 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:25:18.742492 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:25:34.669775 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:25:34.669728 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:25:41.747721 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:25:41.747655 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:25:41.847849 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:25:41.847821 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:25:41.940470 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:25:41.940438 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:25:52.494740 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:25:52.494706 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:03.536753 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:03.536712 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:09.096743 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:09.096708 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:10.720795 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:10.720755 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:10.780447 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:10.780417 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:10.867643 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:10.867613 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:10.962275 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:10.962251 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:11.064393 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:11.064361 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:11.771345 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:11.771308 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:11.885226 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:11.885187 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:11.993199 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:11.993174 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:13.508439 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:13.508408 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:21.839416 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:21.839363 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:21.917968 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:21.917935 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:22.537707 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:22.537674 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:30.610148 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:30.610116 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:30.680764 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:30.680738 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:30.842102 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:30.842074 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:31.181164 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:31.181135 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:31.526861 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:31.526762 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:34.747679 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:34.747616 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:34.832579 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:34.832540 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:35.925753 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:35.925716 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:36.087970 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:36.087933 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:36.745077 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:36.745039 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:36.846961 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:36.846926 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:44.655965 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:44.655904 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:26:59.703491 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:26:59.703448 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:00.330731 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:00.330690 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:00.431401 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:00.431370 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:00.781064 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:00.780985 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:07.232774 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:07.232729 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:07.302566 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:07.302520 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:07.601279 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:07.601244 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6abb689a-qw866"] Aug 04 18:27:07.601606 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:07.601591 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14cab2dc-f0b2-44e7-bbd3-b2d598e465a5" containerName="revoke-keys" Aug 04 18:27:07.601651 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:07.601609 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="14cab2dc-f0b2-44e7-bbd3-b2d598e465a5" containerName="revoke-keys" Aug 04 18:27:07.601756 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:07.601723 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="14cab2dc-f0b2-44e7-bbd3-b2d598e465a5" containerName="revoke-keys" Aug 04 18:27:07.605077 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:07.605057 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6abb689a-qw866" Aug 04 18:27:07.607777 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:07.607751 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pnbq7\"" Aug 04 18:27:07.607897 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:07.607793 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 18:27:07.614742 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:07.614722 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6abb689a-qw866"] Aug 04 18:27:07.738087 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:07.738058 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bc3d128-f49a-460a-8561-7eaa468fb402-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6abb689a-qw866\" (UID: \"0bc3d128-f49a-460a-8561-7eaa468fb402\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6abb689a-qw866" Aug 04 18:27:07.838756 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:07.838729 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bc3d128-f49a-460a-8561-7eaa468fb402-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6abb689a-qw866\" (UID: \"0bc3d128-f49a-460a-8561-7eaa468fb402\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6abb689a-qw866" Aug 04 18:27:07.839295 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:07.839275 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bc3d128-f49a-460a-8561-7eaa468fb402-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6abb689a-qw866\" (UID: \"0bc3d128-f49a-460a-8561-7eaa468fb402\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6abb689a-qw866" Aug 04 18:27:07.915583 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:07.915514 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6abb689a-qw866" Aug 04 18:27:08.057099 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:08.057076 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6abb689a-qw866"] Aug 04 18:27:08.059617 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:27:08.059591 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc3d128_f49a_460a_8561_7eaa468fb402.slice/crio-eacf8271c0cacd0c00042b681b9a10ed472dd81b1f56d5ff45d77d4d5a505f03 WatchSource:0}: Error finding container eacf8271c0cacd0c00042b681b9a10ed472dd81b1f56d5ff45d77d4d5a505f03: Status 404 returned error can't find the container with id eacf8271c0cacd0c00042b681b9a10ed472dd81b1f56d5ff45d77d4d5a505f03 Aug 04 18:27:08.865574 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:08.865542 2572 generic.go:356] "Generic (PLEG): container finished" podID="0bc3d128-f49a-460a-8561-7eaa468fb402" containerID="f7e3179dbc59dfd3b1ff8978118d77f788d4b1b9e8086ff03f066983143453f9" exitCode=0 Aug 04 18:27:08.865981 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:08.865628 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6abb689a-qw866" event={"ID":"0bc3d128-f49a-460a-8561-7eaa468fb402","Type":"ContainerDied","Data":"f7e3179dbc59dfd3b1ff8978118d77f788d4b1b9e8086ff03f066983143453f9"} Aug 04 18:27:08.865981 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:08.865695 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6abb689a-qw866" event={"ID":"0bc3d128-f49a-460a-8561-7eaa468fb402","Type":"ContainerStarted","Data":"eacf8271c0cacd0c00042b681b9a10ed472dd81b1f56d5ff45d77d4d5a505f03"} Aug 04 18:27:09.996060 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:09.996039 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6abb689a-qw866" Aug 04 18:27:10.057376 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:10.057343 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bc3d128-f49a-460a-8561-7eaa468fb402-maas-api-service-ca\") pod \"0bc3d128-f49a-460a-8561-7eaa468fb402\" (UID: \"0bc3d128-f49a-460a-8561-7eaa468fb402\") " Aug 04 18:27:10.057698 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:10.057647 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bc3d128-f49a-460a-8561-7eaa468fb402-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0bc3d128-f49a-460a-8561-7eaa468fb402" (UID: "0bc3d128-f49a-460a-8561-7eaa468fb402"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:27:10.158290 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:10.158225 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bc3d128-f49a-460a-8561-7eaa468fb402-maas-api-service-ca\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:27:10.875683 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:10.875629 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6abb689a-qw866" event={"ID":"0bc3d128-f49a-460a-8561-7eaa468fb402","Type":"ContainerDied","Data":"eacf8271c0cacd0c00042b681b9a10ed472dd81b1f56d5ff45d77d4d5a505f03"} Aug 04 18:27:10.875683 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:10.875685 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6abb689a-qw866" Aug 04 18:27:10.875869 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:10.875708 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eacf8271c0cacd0c00042b681b9a10ed472dd81b1f56d5ff45d77d4d5a505f03" Aug 04 18:27:18.293375 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:18.293339 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:18.418248 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:18.418210 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:18.554291 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:18.554215 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6abb689a-qw866"] Aug 04 18:27:18.559383 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:18.559361 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6abb689a-qw866"] Aug 04 18:27:18.606399 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:18.606371 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:18.677571 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:18.677523 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:18.822278 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:18.822193 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bc3d128-f49a-460a-8561-7eaa468fb402" path="/var/lib/kubelet/pods/0bc3d128-f49a-460a-8561-7eaa468fb402/volumes" Aug 04 18:27:18.926349 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:18.926316 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:25.652834 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:25.652795 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:26.056124 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:26.056082 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:26.158188 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:26.158142 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:26.218786 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:26.218757 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:26.631307 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:26.631258 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:26.695678 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:26.695646 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:33.242195 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:33.242152 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:33.306257 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:33.306226 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:33.418565 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:33.418530 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:33.521213 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:33.521126 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:33.612208 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:33.612174 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:39.880652 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:39.880616 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:27:40.104089 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:27:40.104060 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:29:24.920249 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:29:24.920219 2572 scope.go:117] "RemoveContainer" containerID="d42300ab548bc1627e5bfc9a31de9c892cb2451e2a21c758d8859da00a1f7643" Aug 04 18:29:24.929906 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:29:24.929884 2572 scope.go:117] "RemoveContainer" containerID="e4be6246c56705ac73a5bcc4e8f3522cae5708b03c006d8b05fd754ec3b97b30" Aug 04 18:29:42.177202 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:29:42.177164 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:29:43.290283 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:29:43.290246 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:29:43.514043 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:29:43.514013 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:29:45.828843 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:29:45.828778 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:29:45.928197 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:29:45.928167 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:29:46.980915 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:29:46.980879 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:29:47.291186 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:29:47.291105 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:29:47.990389 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:29:47.990357 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:29:48.090586 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:29:48.090557 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:30:00.133044 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:00.133012 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764470-s2wgq"] Aug 04 18:30:00.133503 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:00.133337 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bc3d128-f49a-460a-8561-7eaa468fb402" containerName="revoke-keys" Aug 04 18:30:00.133503 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:00.133349 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bc3d128-f49a-460a-8561-7eaa468fb402" containerName="revoke-keys" Aug 04 18:30:00.133503 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:00.133410 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bc3d128-f49a-460a-8561-7eaa468fb402" containerName="revoke-keys" Aug 04 18:30:00.136322 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:00.136300 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab5c72f0-29764470-m2p9p"] Aug 04 18:30:00.136449 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:00.136416 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764470-s2wgq" Aug 04 18:30:00.138998 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:00.138979 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pnbq7\"" Aug 04 18:30:00.139375 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:00.139361 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab5c72f0-29764470-m2p9p" Aug 04 18:30:00.144577 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:00.144557 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764470-s2wgq"] Aug 04 18:30:00.147720 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:00.147698 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab5c72f0-29764470-m2p9p"] Aug 04 18:30:00.149369 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:00.149354 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764470-s2wgq" Aug 04 18:30:00.158548 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:00.158530 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab5c72f0-29764470-m2p9p" Aug 04 18:30:00.287756 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:00.287725 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764470-s2wgq"] Aug 04 18:30:00.291240 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:30:00.291213 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c0b408_8066_4900_b5e0_75df9bf2ebf6.slice/crio-be16c9b30e8ca68d657dd9ebc3da3b51d98067106958b34b90ade907f6dd9cdb WatchSource:0}: Error finding container be16c9b30e8ca68d657dd9ebc3da3b51d98067106958b34b90ade907f6dd9cdb: Status 404 returned error can't find the container with id be16c9b30e8ca68d657dd9ebc3da3b51d98067106958b34b90ade907f6dd9cdb Aug 04 18:30:00.309152 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:00.309129 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab5c72f0-29764470-m2p9p"] Aug 04 18:30:00.310793 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:30:00.310771 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259f3100_a47d_4a4f_8736_32071a42a23d.slice/crio-a9ceb35471831226b8925eae900aa0ec54444272d6552072e7bac5797ac0da28 WatchSource:0}: Error finding container a9ceb35471831226b8925eae900aa0ec54444272d6552072e7bac5797ac0da28: Status 404 returned error can't find the container with id a9ceb35471831226b8925eae900aa0ec54444272d6552072e7bac5797ac0da28 Aug 04 18:30:00.641753 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:00.641714 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab5c72f0-29764470-m2p9p" event={"ID":"259f3100-a47d-4a4f-8736-32071a42a23d","Type":"ContainerStarted","Data":"a731b51b9d90b91c532578edb6495f9302cb819882f6c233aaefc6333ae26a45"} Aug 04 18:30:00.641930 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:00.641775 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab5c72f0-29764470-m2p9p" event={"ID":"259f3100-a47d-4a4f-8736-32071a42a23d","Type":"ContainerStarted","Data":"a9ceb35471831226b8925eae900aa0ec54444272d6552072e7bac5797ac0da28"} Aug 04 18:30:00.643193 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:00.643168 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764470-s2wgq" event={"ID":"a1c0b408-8066-4900-b5e0-75df9bf2ebf6","Type":"ContainerStarted","Data":"e7c1452b57cbe8f605f9a532834310093ba798af56a2d884a1abeeb9287110df"} Aug 04 18:30:00.643327 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:00.643206 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764470-s2wgq" event={"ID":"a1c0b408-8066-4900-b5e0-75df9bf2ebf6","Type":"ContainerStarted","Data":"be16c9b30e8ca68d657dd9ebc3da3b51d98067106958b34b90ade907f6dd9cdb"} Aug 04 18:30:00.660715 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:00.660649 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab5c72f0-29764470-m2p9p" podStartSLOduration=0.66063575 podStartE2EDuration="660.63575ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:30:00.659879325 +0000 UTC m=+2256.350570376" watchObservedRunningTime="2026-08-04 18:30:00.66063575 +0000 UTC m=+2256.351326838" Aug 04 18:30:00.676497 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:00.676458 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764470-s2wgq" podStartSLOduration=0.676445842 podStartE2EDuration="676.445842ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:30:00.675476561 +0000 UTC m=+2256.366167613" watchObservedRunningTime="2026-08-04 18:30:00.676445842 +0000 UTC m=+2256.367136893" Aug 04 18:30:01.649480 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:01.649450 2572 generic.go:356] "Generic (PLEG): container finished" podID="a1c0b408-8066-4900-b5e0-75df9bf2ebf6" containerID="e7c1452b57cbe8f605f9a532834310093ba798af56a2d884a1abeeb9287110df" exitCode=0 Aug 04 18:30:01.649939 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:01.649542 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764470-s2wgq" event={"ID":"a1c0b408-8066-4900-b5e0-75df9bf2ebf6","Type":"ContainerDied","Data":"e7c1452b57cbe8f605f9a532834310093ba798af56a2d884a1abeeb9287110df"} Aug 04 18:30:01.651076 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:01.651049 2572 generic.go:356] "Generic (PLEG): container finished" podID="259f3100-a47d-4a4f-8736-32071a42a23d" containerID="a731b51b9d90b91c532578edb6495f9302cb819882f6c233aaefc6333ae26a45" exitCode=0 Aug 04 18:30:01.651203 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:01.651090 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab5c72f0-29764470-m2p9p" event={"ID":"259f3100-a47d-4a4f-8736-32071a42a23d","Type":"ContainerDied","Data":"a731b51b9d90b91c532578edb6495f9302cb819882f6c233aaefc6333ae26a45"} Aug 04 18:30:02.785294 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:02.785269 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab5c72f0-29764470-m2p9p" Aug 04 18:30:02.808189 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:02.808170 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764470-s2wgq" Aug 04 18:30:03.661437 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:03.661406 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab5c72f0-29764470-m2p9p" Aug 04 18:30:03.661437 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:03.661424 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab5c72f0-29764470-m2p9p" event={"ID":"259f3100-a47d-4a4f-8736-32071a42a23d","Type":"ContainerDied","Data":"a9ceb35471831226b8925eae900aa0ec54444272d6552072e7bac5797ac0da28"} Aug 04 18:30:03.661705 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:03.661469 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9ceb35471831226b8925eae900aa0ec54444272d6552072e7bac5797ac0da28" Aug 04 18:30:03.662719 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:03.662697 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764470-s2wgq" Aug 04 18:30:03.662719 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:03.662708 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764470-s2wgq" event={"ID":"a1c0b408-8066-4900-b5e0-75df9bf2ebf6","Type":"ContainerDied","Data":"be16c9b30e8ca68d657dd9ebc3da3b51d98067106958b34b90ade907f6dd9cdb"} Aug 04 18:30:03.662869 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:03.662748 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be16c9b30e8ca68d657dd9ebc3da3b51d98067106958b34b90ade907f6dd9cdb" Aug 04 18:30:10.721832 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:10.721793 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:30:17.133284 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:17.133245 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:30:24.950643 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:24.950613 2572 scope.go:117] "RemoveContainer" containerID="caead1817dac7ea609972cf58ce19f3464a8847003f359f7cf0e579cab522442" Aug 04 18:30:24.958422 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:24.958405 2572 scope.go:117] "RemoveContainer" containerID="00478d1c815290d498487d55bdcbe241f10dd4c4d8e5bed70eafefe1f7a19f69" Aug 04 18:30:30.187479 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.187450 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp"] Aug 04 18:30:30.187932 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.187841 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="259f3100-a47d-4a4f-8736-32071a42a23d" containerName="cleanup" Aug 04 18:30:30.187932 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.187861 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="259f3100-a47d-4a4f-8736-32071a42a23d" containerName="cleanup" Aug 04 18:30:30.187932 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.187884 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1c0b408-8066-4900-b5e0-75df9bf2ebf6" containerName="cleanup" Aug 04 18:30:30.187932 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.187893 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1c0b408-8066-4900-b5e0-75df9bf2ebf6" containerName="cleanup" Aug 04 18:30:30.188094 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.187994 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="259f3100-a47d-4a4f-8736-32071a42a23d" containerName="cleanup" Aug 04 18:30:30.188094 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.188011 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1c0b408-8066-4900-b5e0-75df9bf2ebf6" containerName="cleanup" Aug 04 18:30:30.191024 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.191010 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:30:30.193856 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.193825 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ab5c72f0\"/\"default-dockercfg-6sw5k\"" Aug 04 18:30:30.193993 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.193970 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ab5c72f0\"/\"openshift-service-ca.crt\"" Aug 04 18:30:30.194553 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.194536 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ab5c72f0\"/\"kube-root-ca.crt\"" Aug 04 18:30:30.194640 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.194596 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ab5c72f0\"/\"e2e-lifecycle-model-ab5c72f0-kserve-self-signed-certs\"" Aug 04 18:30:30.201323 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.201298 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp"] Aug 04 18:30:30.234773 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.234736 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:30:30.304472 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.304444 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/591a363c-4c09-4967-bec4-f7d348bcef5f-dshm\") pod \"e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp\" (UID: \"591a363c-4c09-4967-bec4-f7d348bcef5f\") " pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:30:30.304595 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.304477 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg5h7\" (UniqueName: \"kubernetes.io/projected/591a363c-4c09-4967-bec4-f7d348bcef5f-kube-api-access-lg5h7\") pod \"e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp\" (UID: \"591a363c-4c09-4967-bec4-f7d348bcef5f\") " pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:30:30.304595 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.304506 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/591a363c-4c09-4967-bec4-f7d348bcef5f-home\") pod \"e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp\" (UID: \"591a363c-4c09-4967-bec4-f7d348bcef5f\") " pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:30:30.304595 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.304567 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/591a363c-4c09-4967-bec4-f7d348bcef5f-model-cache\") pod \"e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp\" (UID: \"591a363c-4c09-4967-bec4-f7d348bcef5f\") " pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:30:30.304753 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.304608 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/591a363c-4c09-4967-bec4-f7d348bcef5f-tmp-dir\") pod \"e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp\" (UID: \"591a363c-4c09-4967-bec4-f7d348bcef5f\") " pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:30:30.304753 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.304629 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/591a363c-4c09-4967-bec4-f7d348bcef5f-tls-certs\") pod \"e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp\" (UID: \"591a363c-4c09-4967-bec4-f7d348bcef5f\") " pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:30:30.405825 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.405782 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/591a363c-4c09-4967-bec4-f7d348bcef5f-tmp-dir\") pod \"e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp\" (UID: \"591a363c-4c09-4967-bec4-f7d348bcef5f\") " pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:30:30.405825 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.405833 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/591a363c-4c09-4967-bec4-f7d348bcef5f-tls-certs\") pod \"e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp\" (UID: \"591a363c-4c09-4967-bec4-f7d348bcef5f\") " pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:30:30.406047 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.405894 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/591a363c-4c09-4967-bec4-f7d348bcef5f-dshm\") pod \"e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp\" (UID: \"591a363c-4c09-4967-bec4-f7d348bcef5f\") " pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:30:30.406047 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.405922 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg5h7\" (UniqueName: \"kubernetes.io/projected/591a363c-4c09-4967-bec4-f7d348bcef5f-kube-api-access-lg5h7\") pod \"e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp\" (UID: \"591a363c-4c09-4967-bec4-f7d348bcef5f\") " pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:30:30.406047 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.405968 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/591a363c-4c09-4967-bec4-f7d348bcef5f-home\") pod \"e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp\" (UID: \"591a363c-4c09-4967-bec4-f7d348bcef5f\") " pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:30:30.406199 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.406150 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/591a363c-4c09-4967-bec4-f7d348bcef5f-model-cache\") pod \"e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp\" (UID: \"591a363c-4c09-4967-bec4-f7d348bcef5f\") " pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:30:30.406251 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.406219 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/591a363c-4c09-4967-bec4-f7d348bcef5f-tmp-dir\") pod \"e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp\" (UID: \"591a363c-4c09-4967-bec4-f7d348bcef5f\") " pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:30:30.406304 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.406256 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/591a363c-4c09-4967-bec4-f7d348bcef5f-home\") pod \"e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp\" (UID: \"591a363c-4c09-4967-bec4-f7d348bcef5f\") " pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:30:30.406501 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.406481 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/591a363c-4c09-4967-bec4-f7d348bcef5f-model-cache\") pod \"e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp\" (UID: \"591a363c-4c09-4967-bec4-f7d348bcef5f\") " pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:30:30.408369 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.408343 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/591a363c-4c09-4967-bec4-f7d348bcef5f-dshm\") pod \"e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp\" (UID: \"591a363c-4c09-4967-bec4-f7d348bcef5f\") " pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:30:30.408479 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.408460 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/591a363c-4c09-4967-bec4-f7d348bcef5f-tls-certs\") pod \"e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp\" (UID: \"591a363c-4c09-4967-bec4-f7d348bcef5f\") " pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:30:30.418935 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.418912 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg5h7\" (UniqueName: \"kubernetes.io/projected/591a363c-4c09-4967-bec4-f7d348bcef5f-kube-api-access-lg5h7\") pod \"e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp\" (UID: \"591a363c-4c09-4967-bec4-f7d348bcef5f\") " pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:30:30.501053 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.500962 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:30:30.632438 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.632416 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp"] Aug 04 18:30:30.634315 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:30:30.634290 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod591a363c_4c09_4967_bec4_f7d348bcef5f.slice/crio-abb8105a261c3612093877d7ef34565367f38b6142056e7b0eabdf9fb76155f8 WatchSource:0}: Error finding container abb8105a261c3612093877d7ef34565367f38b6142056e7b0eabdf9fb76155f8: Status 404 returned error can't find the container with id abb8105a261c3612093877d7ef34565367f38b6142056e7b0eabdf9fb76155f8 Aug 04 18:30:30.794148 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.794109 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" event={"ID":"591a363c-4c09-4967-bec4-f7d348bcef5f","Type":"ContainerStarted","Data":"dc9e38729b040fbf26e2c3ccef4ad6b557226c2f1724c0e1fd49bd8a72bd869e"} Aug 04 18:30:30.794313 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.794159 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" event={"ID":"591a363c-4c09-4967-bec4-f7d348bcef5f","Type":"ContainerStarted","Data":"abb8105a261c3612093877d7ef34565367f38b6142056e7b0eabdf9fb76155f8"} Aug 04 18:30:30.813465 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:30.813424 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" podStartSLOduration=0.813409046 podStartE2EDuration="813.409046ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:30:30.812549737 +0000 UTC m=+2286.503240789" watchObservedRunningTime="2026-08-04 18:30:30.813409046 +0000 UTC m=+2286.504100096" Aug 04 18:30:40.501771 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:40.501727 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:30:40.502172 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:40.501862 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:30:40.514295 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:40.514256 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:30:40.850138 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:40.850113 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:30:49.646466 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:30:49.646427 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:31:00.630520 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:00.630479 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab5c72f0-qtzwk"] Aug 04 18:31:00.634057 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:00.634035 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab5c72f0-qtzwk" Aug 04 18:31:00.636800 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:00.636770 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 18:31:00.636924 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:00.636863 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pnbq7\"" Aug 04 18:31:00.645983 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:00.645959 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab5c72f0-qtzwk"] Aug 04 18:31:00.738449 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:00.738425 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3128ab20-5593-4791-bed6-17e9f0d24fd1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ab5c72f0-qtzwk\" (UID: \"3128ab20-5593-4791-bed6-17e9f0d24fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab5c72f0-qtzwk" Aug 04 18:31:00.839038 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:00.839006 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3128ab20-5593-4791-bed6-17e9f0d24fd1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ab5c72f0-qtzwk\" (UID: \"3128ab20-5593-4791-bed6-17e9f0d24fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab5c72f0-qtzwk" Aug 04 18:31:00.839597 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:00.839574 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3128ab20-5593-4791-bed6-17e9f0d24fd1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ab5c72f0-qtzwk\" (UID: \"3128ab20-5593-4791-bed6-17e9f0d24fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab5c72f0-qtzwk" Aug 04 18:31:00.945334 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:00.945264 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab5c72f0-qtzwk" Aug 04 18:31:01.077498 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:01.077475 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab5c72f0-qtzwk"] Aug 04 18:31:01.078822 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:31:01.078788 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3128ab20_5593_4791_bed6_17e9f0d24fd1.slice/crio-8b8452ec004e0428a1945e70a5f11f576a5f1fd758739e82d65dd0bf9cef7733 WatchSource:0}: Error finding container 8b8452ec004e0428a1945e70a5f11f576a5f1fd758739e82d65dd0bf9cef7733: Status 404 returned error can't find the container with id 8b8452ec004e0428a1945e70a5f11f576a5f1fd758739e82d65dd0bf9cef7733 Aug 04 18:31:01.934186 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:01.934100 2572 generic.go:356] "Generic (PLEG): container finished" podID="3128ab20-5593-4791-bed6-17e9f0d24fd1" containerID="c4c6a1734dfdafcaa4e357aa8af287bd593c62a757ebf33cbc69be016d6e243a" exitCode=0 Aug 04 18:31:01.934566 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:01.934190 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab5c72f0-qtzwk" event={"ID":"3128ab20-5593-4791-bed6-17e9f0d24fd1","Type":"ContainerDied","Data":"c4c6a1734dfdafcaa4e357aa8af287bd593c62a757ebf33cbc69be016d6e243a"} Aug 04 18:31:01.934566 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:01.934288 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab5c72f0-qtzwk" event={"ID":"3128ab20-5593-4791-bed6-17e9f0d24fd1","Type":"ContainerStarted","Data":"8b8452ec004e0428a1945e70a5f11f576a5f1fd758739e82d65dd0bf9cef7733"} Aug 04 18:31:03.064361 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:03.064340 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab5c72f0-qtzwk" Aug 04 18:31:03.158815 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:03.158786 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3128ab20-5593-4791-bed6-17e9f0d24fd1-maas-api-service-ca\") pod \"3128ab20-5593-4791-bed6-17e9f0d24fd1\" (UID: \"3128ab20-5593-4791-bed6-17e9f0d24fd1\") " Aug 04 18:31:03.159104 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:03.159084 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3128ab20-5593-4791-bed6-17e9f0d24fd1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3128ab20-5593-4791-bed6-17e9f0d24fd1" (UID: "3128ab20-5593-4791-bed6-17e9f0d24fd1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:31:03.260115 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:03.260034 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3128ab20-5593-4791-bed6-17e9f0d24fd1-maas-api-service-ca\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:31:03.943679 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:03.943638 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab5c72f0-qtzwk" Aug 04 18:31:03.943679 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:03.943627 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab5c72f0-qtzwk" event={"ID":"3128ab20-5593-4791-bed6-17e9f0d24fd1","Type":"ContainerDied","Data":"8b8452ec004e0428a1945e70a5f11f576a5f1fd758739e82d65dd0bf9cef7733"} Aug 04 18:31:03.943890 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:03.943703 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b8452ec004e0428a1945e70a5f11f576a5f1fd758739e82d65dd0bf9cef7733" Aug 04 18:31:16.294684 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:16.294583 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab5c72f0-29764470-m2p9p"] Aug 04 18:31:16.302813 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:16.302790 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-ab5c72f0-29764470-m2p9p"] Aug 04 18:31:16.309256 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:16.309232 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:31:16.384801 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:16.384755 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:31:16.476747 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:16.476717 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:31:16.541694 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:16.541639 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab5c72f0-qtzwk"] Aug 04 18:31:16.544059 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:16.544035 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ab5c72f0-qtzwk"] Aug 04 18:31:16.545943 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:16.545880 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:31:16.821020 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:16.820944 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="259f3100-a47d-4a4f-8736-32071a42a23d" path="/var/lib/kubelet/pods/259f3100-a47d-4a4f-8736-32071a42a23d/volumes" Aug 04 18:31:16.821248 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:16.821234 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3128ab20-5593-4791-bed6-17e9f0d24fd1" path="/var/lib/kubelet/pods/3128ab20-5593-4791-bed6-17e9f0d24fd1/volumes" Aug 04 18:31:19.731972 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:19.731940 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:31:24.715300 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:24.715265 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp"] Aug 04 18:31:24.811115 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:24.811080 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:31:39.734752 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:39.734705 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" podUID="591a363c-4c09-4967-bec4-f7d348bcef5f" containerName="main" containerID="cri-o://dc9e38729b040fbf26e2c3ccef4ad6b557226c2f1724c0e1fd49bd8a72bd869e" gracePeriod=45 Aug 04 18:31:39.977176 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:39.977147 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:31:40.112286 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.112253 2572 generic.go:356] "Generic (PLEG): container finished" podID="591a363c-4c09-4967-bec4-f7d348bcef5f" containerID="dc9e38729b040fbf26e2c3ccef4ad6b557226c2f1724c0e1fd49bd8a72bd869e" exitCode=0 Aug 04 18:31:40.112441 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.112297 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" Aug 04 18:31:40.112441 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.112335 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" event={"ID":"591a363c-4c09-4967-bec4-f7d348bcef5f","Type":"ContainerDied","Data":"dc9e38729b040fbf26e2c3ccef4ad6b557226c2f1724c0e1fd49bd8a72bd869e"} Aug 04 18:31:40.112441 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.112382 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp" event={"ID":"591a363c-4c09-4967-bec4-f7d348bcef5f","Type":"ContainerDied","Data":"abb8105a261c3612093877d7ef34565367f38b6142056e7b0eabdf9fb76155f8"} Aug 04 18:31:40.112441 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.112405 2572 scope.go:117] "RemoveContainer" containerID="dc9e38729b040fbf26e2c3ccef4ad6b557226c2f1724c0e1fd49bd8a72bd869e" Aug 04 18:31:40.122580 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.122543 2572 scope.go:117] "RemoveContainer" containerID="dc9e38729b040fbf26e2c3ccef4ad6b557226c2f1724c0e1fd49bd8a72bd869e" Aug 04 18:31:40.122986 ip-10-0-137-173 kubenswrapper[2572]: E0804 18:31:40.122963 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc9e38729b040fbf26e2c3ccef4ad6b557226c2f1724c0e1fd49bd8a72bd869e\": container with ID starting with dc9e38729b040fbf26e2c3ccef4ad6b557226c2f1724c0e1fd49bd8a72bd869e not found: ID does not exist" containerID="dc9e38729b040fbf26e2c3ccef4ad6b557226c2f1724c0e1fd49bd8a72bd869e" Aug 04 18:31:40.123086 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.123008 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc9e38729b040fbf26e2c3ccef4ad6b557226c2f1724c0e1fd49bd8a72bd869e"} err="failed to get container status \"dc9e38729b040fbf26e2c3ccef4ad6b557226c2f1724c0e1fd49bd8a72bd869e\": rpc error: code = NotFound desc = could not find container \"dc9e38729b040fbf26e2c3ccef4ad6b557226c2f1724c0e1fd49bd8a72bd869e\": container with ID starting with dc9e38729b040fbf26e2c3ccef4ad6b557226c2f1724c0e1fd49bd8a72bd869e not found: ID does not exist" Aug 04 18:31:40.169908 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.169879 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/591a363c-4c09-4967-bec4-f7d348bcef5f-dshm\") pod \"591a363c-4c09-4967-bec4-f7d348bcef5f\" (UID: \"591a363c-4c09-4967-bec4-f7d348bcef5f\") " Aug 04 18:31:40.170033 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.169944 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/591a363c-4c09-4967-bec4-f7d348bcef5f-home\") pod \"591a363c-4c09-4967-bec4-f7d348bcef5f\" (UID: \"591a363c-4c09-4967-bec4-f7d348bcef5f\") " Aug 04 18:31:40.170033 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.170002 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/591a363c-4c09-4967-bec4-f7d348bcef5f-tmp-dir\") pod \"591a363c-4c09-4967-bec4-f7d348bcef5f\" (UID: \"591a363c-4c09-4967-bec4-f7d348bcef5f\") " Aug 04 18:31:40.170149 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.170039 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lg5h7\" (UniqueName: \"kubernetes.io/projected/591a363c-4c09-4967-bec4-f7d348bcef5f-kube-api-access-lg5h7\") pod \"591a363c-4c09-4967-bec4-f7d348bcef5f\" (UID: \"591a363c-4c09-4967-bec4-f7d348bcef5f\") " Aug 04 18:31:40.170149 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.170083 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/591a363c-4c09-4967-bec4-f7d348bcef5f-tls-certs\") pod \"591a363c-4c09-4967-bec4-f7d348bcef5f\" (UID: \"591a363c-4c09-4967-bec4-f7d348bcef5f\") " Aug 04 18:31:40.170149 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.170113 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/591a363c-4c09-4967-bec4-f7d348bcef5f-model-cache\") pod \"591a363c-4c09-4967-bec4-f7d348bcef5f\" (UID: \"591a363c-4c09-4967-bec4-f7d348bcef5f\") " Aug 04 18:31:40.170291 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.170223 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591a363c-4c09-4967-bec4-f7d348bcef5f-home" (OuterVolumeSpecName: "home") pod "591a363c-4c09-4967-bec4-f7d348bcef5f" (UID: "591a363c-4c09-4967-bec4-f7d348bcef5f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:31:40.170344 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.170314 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591a363c-4c09-4967-bec4-f7d348bcef5f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "591a363c-4c09-4967-bec4-f7d348bcef5f" (UID: "591a363c-4c09-4967-bec4-f7d348bcef5f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:31:40.170422 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.170395 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591a363c-4c09-4967-bec4-f7d348bcef5f-model-cache" (OuterVolumeSpecName: "model-cache") pod "591a363c-4c09-4967-bec4-f7d348bcef5f" (UID: "591a363c-4c09-4967-bec4-f7d348bcef5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:31:40.170488 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.170469 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/591a363c-4c09-4967-bec4-f7d348bcef5f-home\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:31:40.170534 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.170494 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/591a363c-4c09-4967-bec4-f7d348bcef5f-tmp-dir\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:31:40.172259 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.172236 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/591a363c-4c09-4967-bec4-f7d348bcef5f-dshm" (OuterVolumeSpecName: "dshm") pod "591a363c-4c09-4967-bec4-f7d348bcef5f" (UID: "591a363c-4c09-4967-bec4-f7d348bcef5f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:31:40.172357 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.172273 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/591a363c-4c09-4967-bec4-f7d348bcef5f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "591a363c-4c09-4967-bec4-f7d348bcef5f" (UID: "591a363c-4c09-4967-bec4-f7d348bcef5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:31:40.172357 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.172284 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591a363c-4c09-4967-bec4-f7d348bcef5f-kube-api-access-lg5h7" (OuterVolumeSpecName: "kube-api-access-lg5h7") pod "591a363c-4c09-4967-bec4-f7d348bcef5f" (UID: "591a363c-4c09-4967-bec4-f7d348bcef5f"). InnerVolumeSpecName "kube-api-access-lg5h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:31:40.271444 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.271421 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/591a363c-4c09-4967-bec4-f7d348bcef5f-tls-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:31:40.271444 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.271444 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/591a363c-4c09-4967-bec4-f7d348bcef5f-model-cache\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:31:40.271590 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.271452 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/591a363c-4c09-4967-bec4-f7d348bcef5f-dshm\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:31:40.271590 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.271462 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg5h7\" (UniqueName: \"kubernetes.io/projected/591a363c-4c09-4967-bec4-f7d348bcef5f-kube-api-access-lg5h7\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:31:40.434390 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.434363 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp"] Aug 04 18:31:40.439243 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.439219 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-ab5c72f0/e2e-lifecycle-model-ab5c72f0-kserve-85f8cd47ff-m74tp"] Aug 04 18:31:40.822445 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:31:40.822413 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="591a363c-4c09-4967-bec4-f7d348bcef5f" path="/var/lib/kubelet/pods/591a363c-4c09-4967-bec4-f7d348bcef5f/volumes" Aug 04 18:32:01.653435 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:32:01.653388 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:32:02.415705 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:32:02.415654 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:32:08.974196 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:32:08.974158 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:32:09.042909 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:32:09.042880 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:32:09.255850 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:32:09.255769 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:32:09.493049 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:32:09.493011 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:32:20.811064 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:32:20.811026 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:32:20.917237 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:32:20.917196 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:32:21.164529 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:32:21.164447 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:32:21.591762 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:32:21.591721 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:32:21.689567 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:32:21.689535 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:32:21.789990 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:32:21.789956 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:32:23.024827 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:32:23.024800 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:32:30.276509 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:32:30.276473 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:32:30.350184 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:32:30.350153 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:32:30.966628 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:32:30.966596 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:32:38.662730 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:32:38.662693 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:32:38.775025 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:32:38.774995 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:32:48.234933 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:32:48.234899 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:32:48.939983 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:32:48.939946 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:32:49.200956 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:32:49.200875 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:33:10.139276 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:10.139238 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:33:10.978590 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:10.978551 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:33:11.081456 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:11.081423 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:33:16.942243 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:16.942206 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:33:17.001745 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:17.001712 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:33:17.348853 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:17.348815 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:33:17.678934 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:17.678856 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:33:18.182966 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:18.182933 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:33:21.096130 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:21.096093 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:33:21.195776 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:21.195744 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:33:22.230548 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:22.230513 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:33:22.500031 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:22.499948 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:33:22.790441 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:22.790362 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:33:22.890548 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:22.890516 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:33:24.997037 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:24.997009 2572 scope.go:117] "RemoveContainer" containerID="f7e3179dbc59dfd3b1ff8978118d77f788d4b1b9e8086ff03f066983143453f9" Aug 04 18:33:46.017559 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:46.017476 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:33:46.362232 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:46.362198 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:33:46.802658 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:46.802619 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:33:46.806991 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:46.806965 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s"] Aug 04 18:33:46.807313 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:46.807301 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3128ab20-5593-4791-bed6-17e9f0d24fd1" containerName="revoke-keys" Aug 04 18:33:46.807356 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:46.807316 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3128ab20-5593-4791-bed6-17e9f0d24fd1" containerName="revoke-keys" Aug 04 18:33:46.807356 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:46.807327 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="591a363c-4c09-4967-bec4-f7d348bcef5f" containerName="main" Aug 04 18:33:46.807356 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:46.807332 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="591a363c-4c09-4967-bec4-f7d348bcef5f" containerName="main" Aug 04 18:33:46.807503 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:46.807395 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3128ab20-5593-4791-bed6-17e9f0d24fd1" containerName="revoke-keys" Aug 04 18:33:46.807503 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:46.807403 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="591a363c-4c09-4967-bec4-f7d348bcef5f" containerName="main" Aug 04 18:33:46.810289 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:46.810275 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:46.814633 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:46.814610 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7f7c0a-openshift-default-dockercfg-nxdw5\"" Aug 04 18:33:46.836867 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:46.836842 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s"] Aug 04 18:33:46.919245 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:46.919217 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istio-envoy\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:46.919428 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:46.919263 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istio-podinfo\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:46.919428 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:46.919285 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istiod-ca-cert\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:46.919428 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:46.919307 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-workload-certs\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:46.919428 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:46.919323 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88drt\" (UniqueName: \"kubernetes.io/projected/46544f7d-0eca-4aa2-83ab-86aea841ecf6-kube-api-access-88drt\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:46.919428 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:46.919348 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istio-token\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:46.919428 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:46.919370 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-workload-socket\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:46.919428 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:46.919386 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-credential-socket\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:46.919428 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:46.919420 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istio-data\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:47.020401 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:47.020374 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istio-data\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:47.020848 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:47.020414 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istio-envoy\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:47.020848 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:47.020446 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istio-podinfo\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:47.020848 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:47.020464 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istiod-ca-cert\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:47.020848 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:47.020485 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-workload-certs\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:47.020848 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:47.020501 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88drt\" (UniqueName: \"kubernetes.io/projected/46544f7d-0eca-4aa2-83ab-86aea841ecf6-kube-api-access-88drt\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:47.020848 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:47.020527 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istio-token\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:47.020848 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:47.020630 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-workload-socket\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:47.020848 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:47.020699 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-credential-socket\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:47.021229 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:47.020849 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istio-data\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:47.021229 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:47.020963 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-workload-certs\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:47.021229 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:47.021035 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-credential-socket\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:47.021229 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:47.021078 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-workload-socket\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:47.021229 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:47.021195 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istiod-ca-cert\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:47.022965 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:47.022948 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istio-envoy\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:47.023066 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:47.023051 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istio-podinfo\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:47.037104 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:47.037081 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istio-token\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:47.056893 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:47.056830 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:33:47.061429 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:47.061412 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88drt\" (UniqueName: \"kubernetes.io/projected/46544f7d-0eca-4aa2-83ab-86aea841ecf6-kube-api-access-88drt\") pod \"e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:47.120583 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:47.120548 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:47.268462 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:47.268432 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s"] Aug 04 18:33:47.268788 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:33:47.268763 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46544f7d_0eca_4aa2_83ab_86aea841ecf6.slice/crio-b2ccf9af1c8e7b2767a082b12869e95eeca118d4f31c900fb28ecdb9cdd30b33 WatchSource:0}: Error finding container b2ccf9af1c8e7b2767a082b12869e95eeca118d4f31c900fb28ecdb9cdd30b33: Status 404 returned error can't find the container with id b2ccf9af1c8e7b2767a082b12869e95eeca118d4f31c900fb28ecdb9cdd30b33 Aug 04 18:33:47.270695 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:47.270658 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 18:33:47.680280 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:47.680247 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" event={"ID":"46544f7d-0eca-4aa2-83ab-86aea841ecf6","Type":"ContainerStarted","Data":"b2ccf9af1c8e7b2767a082b12869e95eeca118d4f31c900fb28ecdb9cdd30b33"} Aug 04 18:33:50.389110 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:50.389072 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:33:50.389497 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:50.389158 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:33:50.389497 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:50.389208 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:33:50.696275 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:50.696195 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" event={"ID":"46544f7d-0eca-4aa2-83ab-86aea841ecf6","Type":"ContainerStarted","Data":"258c6b88b1ae43de4bd5e44aaecdc77ceb4d79d8cf11cd400a81a76bb8db6543"} Aug 04 18:33:50.719217 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:50.719172 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" podStartSLOduration=1.601197354 podStartE2EDuration="4.719156795s" totalImagesPullingTime="3.117959441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:33:50.718079312 +0000 UTC m=+2486.408770364" watchObservedRunningTime="2026-08-04 18:33:50.719156795 +0000 UTC m=+2486.409847845" Aug 04 18:33:51.121398 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:51.121367 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:51.126174 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:51.126143 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:51.701044 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:51.701010 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:51.701979 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:51.701959 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:33:52.676920 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:52.676882 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:33:52.777981 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:52.777929 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:33:53.820134 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:53.820102 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:33:54.014121 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:54.014081 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:33:54.349342 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:54.349293 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:33:54.459029 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:33:54.458971 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:34:05.778682 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:05.778592 2572 generic.go:356] "Generic (PLEG): container finished" podID="46544f7d-0eca-4aa2-83ab-86aea841ecf6" containerID="258c6b88b1ae43de4bd5e44aaecdc77ceb4d79d8cf11cd400a81a76bb8db6543" exitCode=0 Aug 04 18:34:05.778682 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:05.778655 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" event={"ID":"46544f7d-0eca-4aa2-83ab-86aea841ecf6","Type":"ContainerDied","Data":"258c6b88b1ae43de4bd5e44aaecdc77ceb4d79d8cf11cd400a81a76bb8db6543"} Aug 04 18:34:05.779125 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:05.779031 2572 scope.go:117] "RemoveContainer" containerID="258c6b88b1ae43de4bd5e44aaecdc77ceb4d79d8cf11cd400a81a76bb8db6543" Aug 04 18:34:05.780140 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:05.780110 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:34:05.780206 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:05.780181 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:34:05.780250 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:05.780213 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:34:06.701242 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:06.701200 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:34:06.787535 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:06.787496 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" event={"ID":"46544f7d-0eca-4aa2-83ab-86aea841ecf6","Type":"ContainerStarted","Data":"6628696e25c261abcab80a62ae5a5f7b4e6d16189747b38aba4bdc182e1275c8"} Aug 04 18:34:07.121336 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:07.121296 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:34:07.122491 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:07.122457 2572 patch_prober.go:28] interesting pod/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Aug 04 18:34:07.122629 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:07.122510 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" podUID="46544f7d-0eca-4aa2-83ab-86aea841ecf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Aug 04 18:34:08.121742 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:08.121709 2572 patch_prober.go:28] interesting pod/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Aug 04 18:34:08.122105 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:08.121768 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" podUID="46544f7d-0eca-4aa2-83ab-86aea841ecf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Aug 04 18:34:09.121489 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:09.121457 2572 patch_prober.go:28] interesting pod/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Aug 04 18:34:09.121690 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:09.121512 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" podUID="46544f7d-0eca-4aa2-83ab-86aea841ecf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Aug 04 18:34:10.121463 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:10.121425 2572 patch_prober.go:28] interesting pod/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Aug 04 18:34:10.121894 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:10.121483 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" podUID="46544f7d-0eca-4aa2-83ab-86aea841ecf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Aug 04 18:34:11.121497 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:11.121462 2572 patch_prober.go:28] interesting pod/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Aug 04 18:34:11.121902 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:11.121545 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" podUID="46544f7d-0eca-4aa2-83ab-86aea841ecf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Aug 04 18:34:12.125574 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:12.125544 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:34:12.126096 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:12.125921 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:34:12.126452 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:12.126437 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:34:22.761071 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:22.760974 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:34:34.793038 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:34.793006 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf"] Aug 04 18:34:34.795937 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:34.795922 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:34:34.798700 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:34.798653 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c70eb3\"/\"openshift-service-ca.crt\"" Aug 04 18:34:34.799470 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:34.799455 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c70eb3\"/\"test-model-c70eb3-kserve-self-signed-certs\"" Aug 04 18:34:34.799620 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:34.799602 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c70eb3\"/\"default-dockercfg-75wtb\"" Aug 04 18:34:34.799723 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:34.799708 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c70eb3\"/\"kube-root-ca.crt\"" Aug 04 18:34:34.806323 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:34.806303 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf"] Aug 04 18:34:34.820199 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:34.820174 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:34:34.948745 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:34.948719 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-home\") pod \"test-model-c70eb3-kserve-6f6855d9cc-dh4tf\" (UID: \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\") " pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:34:34.948916 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:34.948776 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-tls-certs\") pod \"test-model-c70eb3-kserve-6f6855d9cc-dh4tf\" (UID: \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\") " pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:34:34.948916 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:34.948824 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-tmp-dir\") pod \"test-model-c70eb3-kserve-6f6855d9cc-dh4tf\" (UID: \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\") " pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:34:34.948916 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:34.948846 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-model-cache\") pod \"test-model-c70eb3-kserve-6f6855d9cc-dh4tf\" (UID: \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\") " pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:34:34.948916 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:34.948895 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqmps\" (UniqueName: \"kubernetes.io/projected/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-kube-api-access-dqmps\") pod \"test-model-c70eb3-kserve-6f6855d9cc-dh4tf\" (UID: \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\") " pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:34:34.949072 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:34.948923 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-dshm\") pod \"test-model-c70eb3-kserve-6f6855d9cc-dh4tf\" (UID: \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\") " pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:34:35.049940 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:35.049899 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-tmp-dir\") pod \"test-model-c70eb3-kserve-6f6855d9cc-dh4tf\" (UID: \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\") " pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:34:35.050118 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:35.049948 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-model-cache\") pod \"test-model-c70eb3-kserve-6f6855d9cc-dh4tf\" (UID: \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\") " pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:34:35.050118 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:35.049976 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqmps\" (UniqueName: \"kubernetes.io/projected/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-kube-api-access-dqmps\") pod \"test-model-c70eb3-kserve-6f6855d9cc-dh4tf\" (UID: \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\") " pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:34:35.050118 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:35.050001 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-dshm\") pod \"test-model-c70eb3-kserve-6f6855d9cc-dh4tf\" (UID: \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\") " pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:34:35.050289 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:35.050154 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-home\") pod \"test-model-c70eb3-kserve-6f6855d9cc-dh4tf\" (UID: \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\") " pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:34:35.050289 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:35.050259 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-tls-certs\") pod \"test-model-c70eb3-kserve-6f6855d9cc-dh4tf\" (UID: \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\") " pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:34:35.050392 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:35.050339 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-tmp-dir\") pod \"test-model-c70eb3-kserve-6f6855d9cc-dh4tf\" (UID: \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\") " pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:34:35.050392 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:35.050368 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-model-cache\") pod \"test-model-c70eb3-kserve-6f6855d9cc-dh4tf\" (UID: \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\") " pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:34:35.050514 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:35.050496 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-home\") pod \"test-model-c70eb3-kserve-6f6855d9cc-dh4tf\" (UID: \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\") " pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:34:35.052421 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:35.052402 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-dshm\") pod \"test-model-c70eb3-kserve-6f6855d9cc-dh4tf\" (UID: \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\") " pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:34:35.052680 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:35.052650 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-tls-certs\") pod \"test-model-c70eb3-kserve-6f6855d9cc-dh4tf\" (UID: \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\") " pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:34:35.060210 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:35.060189 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqmps\" (UniqueName: \"kubernetes.io/projected/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-kube-api-access-dqmps\") pod \"test-model-c70eb3-kserve-6f6855d9cc-dh4tf\" (UID: \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\") " pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:34:35.105184 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:35.105163 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:34:35.235921 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:35.235896 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf"] Aug 04 18:34:35.238039 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:34:35.238016 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bb89e7b_26cc_47a4_85bd_c38a15e3c150.slice/crio-50f96e79791cae255d75b0e4552669fbc086c114c8013a696f5fd308778e00f1 WatchSource:0}: Error finding container 50f96e79791cae255d75b0e4552669fbc086c114c8013a696f5fd308778e00f1: Status 404 returned error can't find the container with id 50f96e79791cae255d75b0e4552669fbc086c114c8013a696f5fd308778e00f1 Aug 04 18:34:35.923206 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:35.923172 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" event={"ID":"4bb89e7b-26cc-47a4-85bd-c38a15e3c150","Type":"ContainerStarted","Data":"9046e14cdf77fc5caa7f97d553d25db5f153806da8b1bbeeedba4705f57fcef4"} Aug 04 18:34:35.923580 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:35.923216 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" event={"ID":"4bb89e7b-26cc-47a4-85bd-c38a15e3c150","Type":"ContainerStarted","Data":"50f96e79791cae255d75b0e4552669fbc086c114c8013a696f5fd308778e00f1"} Aug 04 18:34:35.960144 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:35.960093 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" podStartSLOduration=1.960077708 podStartE2EDuration="1.960077708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:34:35.959466875 +0000 UTC m=+2531.650157950" watchObservedRunningTime="2026-08-04 18:34:35.960077708 +0000 UTC m=+2531.650768759" Aug 04 18:34:45.105312 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:45.105278 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:34:45.105877 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:45.105346 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:34:45.117926 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:45.117906 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:34:45.980179 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:34:45.980091 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:35:11.074118 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.074082 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2"] Aug 04 18:35:11.077592 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.077573 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:35:11.083047 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.083025 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7f7c0a\"/\"openshift-service-ca.crt\"" Aug 04 18:35:11.083162 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.083031 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7f7c0a\"/\"default-dockercfg-ltddz\"" Aug 04 18:35:11.083986 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.083966 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7f7c0a\"/\"test-model-7f7c0a-kserve-self-signed-certs\"" Aug 04 18:35:11.084062 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.084018 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7f7c0a\"/\"kube-root-ca.crt\"" Aug 04 18:35:11.092385 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.092361 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2"] Aug 04 18:35:11.098595 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.098574 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:35:11.148458 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.148430 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68c82a75-6938-4e79-9be1-b734b6840877-tls-certs\") pod \"test-model-7f7c0a-kserve-656b6477f7-sshx2\" (UID: \"68c82a75-6938-4e79-9be1-b734b6840877\") " pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:35:11.148607 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.148482 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68c82a75-6938-4e79-9be1-b734b6840877-home\") pod \"test-model-7f7c0a-kserve-656b6477f7-sshx2\" (UID: \"68c82a75-6938-4e79-9be1-b734b6840877\") " pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:35:11.148607 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.148500 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c82a75-6938-4e79-9be1-b734b6840877-dshm\") pod \"test-model-7f7c0a-kserve-656b6477f7-sshx2\" (UID: \"68c82a75-6938-4e79-9be1-b734b6840877\") " pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:35:11.148723 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.148599 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c82a75-6938-4e79-9be1-b734b6840877-model-cache\") pod \"test-model-7f7c0a-kserve-656b6477f7-sshx2\" (UID: \"68c82a75-6938-4e79-9be1-b734b6840877\") " pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:35:11.148723 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.148626 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c82a75-6938-4e79-9be1-b734b6840877-tmp-dir\") pod \"test-model-7f7c0a-kserve-656b6477f7-sshx2\" (UID: \"68c82a75-6938-4e79-9be1-b734b6840877\") " pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:35:11.148723 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.148645 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcpd2\" (UniqueName: \"kubernetes.io/projected/68c82a75-6938-4e79-9be1-b734b6840877-kube-api-access-xcpd2\") pod \"test-model-7f7c0a-kserve-656b6477f7-sshx2\" (UID: \"68c82a75-6938-4e79-9be1-b734b6840877\") " pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:35:11.249444 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.249397 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68c82a75-6938-4e79-9be1-b734b6840877-home\") pod \"test-model-7f7c0a-kserve-656b6477f7-sshx2\" (UID: \"68c82a75-6938-4e79-9be1-b734b6840877\") " pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:35:11.249444 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.249430 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c82a75-6938-4e79-9be1-b734b6840877-dshm\") pod \"test-model-7f7c0a-kserve-656b6477f7-sshx2\" (UID: \"68c82a75-6938-4e79-9be1-b734b6840877\") " pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:35:11.249717 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.249463 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c82a75-6938-4e79-9be1-b734b6840877-model-cache\") pod \"test-model-7f7c0a-kserve-656b6477f7-sshx2\" (UID: \"68c82a75-6938-4e79-9be1-b734b6840877\") " pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:35:11.249717 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.249689 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c82a75-6938-4e79-9be1-b734b6840877-tmp-dir\") pod \"test-model-7f7c0a-kserve-656b6477f7-sshx2\" (UID: \"68c82a75-6938-4e79-9be1-b734b6840877\") " pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:35:11.249818 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.249735 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcpd2\" (UniqueName: \"kubernetes.io/projected/68c82a75-6938-4e79-9be1-b734b6840877-kube-api-access-xcpd2\") pod \"test-model-7f7c0a-kserve-656b6477f7-sshx2\" (UID: \"68c82a75-6938-4e79-9be1-b734b6840877\") " pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:35:11.249818 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.249793 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68c82a75-6938-4e79-9be1-b734b6840877-tls-certs\") pod \"test-model-7f7c0a-kserve-656b6477f7-sshx2\" (UID: \"68c82a75-6938-4e79-9be1-b734b6840877\") " pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:35:11.249973 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.249943 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c82a75-6938-4e79-9be1-b734b6840877-model-cache\") pod \"test-model-7f7c0a-kserve-656b6477f7-sshx2\" (UID: \"68c82a75-6938-4e79-9be1-b734b6840877\") " pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:35:11.249973 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.249942 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68c82a75-6938-4e79-9be1-b734b6840877-home\") pod \"test-model-7f7c0a-kserve-656b6477f7-sshx2\" (UID: \"68c82a75-6938-4e79-9be1-b734b6840877\") " pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:35:11.250165 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.250084 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c82a75-6938-4e79-9be1-b734b6840877-tmp-dir\") pod \"test-model-7f7c0a-kserve-656b6477f7-sshx2\" (UID: \"68c82a75-6938-4e79-9be1-b734b6840877\") " pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:35:11.252095 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.252066 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c82a75-6938-4e79-9be1-b734b6840877-dshm\") pod \"test-model-7f7c0a-kserve-656b6477f7-sshx2\" (UID: \"68c82a75-6938-4e79-9be1-b734b6840877\") " pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:35:11.252316 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.252301 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68c82a75-6938-4e79-9be1-b734b6840877-tls-certs\") pod \"test-model-7f7c0a-kserve-656b6477f7-sshx2\" (UID: \"68c82a75-6938-4e79-9be1-b734b6840877\") " pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:35:11.263291 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.263273 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcpd2\" (UniqueName: \"kubernetes.io/projected/68c82a75-6938-4e79-9be1-b734b6840877-kube-api-access-xcpd2\") pod \"test-model-7f7c0a-kserve-656b6477f7-sshx2\" (UID: \"68c82a75-6938-4e79-9be1-b734b6840877\") " pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:35:11.387224 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.387139 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:35:11.519206 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:11.519178 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2"] Aug 04 18:35:11.520915 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:35:11.520892 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68c82a75_6938_4e79_9be1_b734b6840877.slice/crio-3b4c70623900f91daa2ecbaa3361c9361ae4e3d00bc386df0ea83faf819ee671 WatchSource:0}: Error finding container 3b4c70623900f91daa2ecbaa3361c9361ae4e3d00bc386df0ea83faf819ee671: Status 404 returned error can't find the container with id 3b4c70623900f91daa2ecbaa3361c9361ae4e3d00bc386df0ea83faf819ee671 Aug 04 18:35:12.086752 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:12.086711 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" event={"ID":"68c82a75-6938-4e79-9be1-b734b6840877","Type":"ContainerStarted","Data":"b2d98b18c1b5fd7095e6178c438f5fbfcbe022cd2a71e021c6608025ded70627"} Aug 04 18:35:12.086752 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:12.086760 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" event={"ID":"68c82a75-6938-4e79-9be1-b734b6840877","Type":"ContainerStarted","Data":"3b4c70623900f91daa2ecbaa3361c9361ae4e3d00bc386df0ea83faf819ee671"} Aug 04 18:35:12.105343 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:12.105217 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" podStartSLOduration=1.105202217 podStartE2EDuration="1.105202217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:35:12.104684302 +0000 UTC m=+2567.795375343" watchObservedRunningTime="2026-08-04 18:35:12.105202217 +0000 UTC m=+2567.795893268" Aug 04 18:35:21.387777 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:21.387737 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:35:21.388393 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:21.387801 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:35:21.400245 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:21.400222 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:35:22.145156 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:22.145126 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:35:25.832029 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:25.831990 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:35:25.890255 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:25.890230 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:35:52.727235 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:52.727203 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf"] Aug 04 18:35:52.762340 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:52.762312 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:35:53.772323 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:53.772285 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2"] Aug 04 18:35:53.794171 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:35:53.794144 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:04.791418 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:04.791382 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:05.600237 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:05.600198 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:05.660253 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:05.660217 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:07.746372 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:07.746306 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" podUID="4bb89e7b-26cc-47a4-85bd-c38a15e3c150" containerName="main" containerID="cri-o://9046e14cdf77fc5caa7f97d553d25db5f153806da8b1bbeeedba4705f57fcef4" gracePeriod=45 Aug 04 18:36:07.979603 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:07.979580 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:36:08.038503 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.038425 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-dshm\") pod \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\" (UID: \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\") " Aug 04 18:36:08.038691 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.038512 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqmps\" (UniqueName: \"kubernetes.io/projected/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-kube-api-access-dqmps\") pod \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\" (UID: \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\") " Aug 04 18:36:08.038691 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.038555 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-model-cache\") pod \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\" (UID: \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\") " Aug 04 18:36:08.038691 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.038583 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-tmp-dir\") pod \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\" (UID: \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\") " Aug 04 18:36:08.038691 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.038629 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-tls-certs\") pod \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\" (UID: \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\") " Aug 04 18:36:08.038691 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.038650 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-home\") pod \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\" (UID: \"4bb89e7b-26cc-47a4-85bd-c38a15e3c150\") " Aug 04 18:36:08.038987 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.038820 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-model-cache" (OuterVolumeSpecName: "model-cache") pod "4bb89e7b-26cc-47a4-85bd-c38a15e3c150" (UID: "4bb89e7b-26cc-47a4-85bd-c38a15e3c150"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:36:08.038987 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.038888 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4bb89e7b-26cc-47a4-85bd-c38a15e3c150" (UID: "4bb89e7b-26cc-47a4-85bd-c38a15e3c150"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:36:08.038987 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.038959 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-model-cache\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:36:08.039315 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.038979 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-home" (OuterVolumeSpecName: "home") pod "4bb89e7b-26cc-47a4-85bd-c38a15e3c150" (UID: "4bb89e7b-26cc-47a4-85bd-c38a15e3c150"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:36:08.040812 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.040788 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-dshm" (OuterVolumeSpecName: "dshm") pod "4bb89e7b-26cc-47a4-85bd-c38a15e3c150" (UID: "4bb89e7b-26cc-47a4-85bd-c38a15e3c150"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:36:08.040812 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.040804 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4bb89e7b-26cc-47a4-85bd-c38a15e3c150" (UID: "4bb89e7b-26cc-47a4-85bd-c38a15e3c150"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:36:08.040959 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.040890 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-kube-api-access-dqmps" (OuterVolumeSpecName: "kube-api-access-dqmps") pod "4bb89e7b-26cc-47a4-85bd-c38a15e3c150" (UID: "4bb89e7b-26cc-47a4-85bd-c38a15e3c150"). InnerVolumeSpecName "kube-api-access-dqmps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:36:08.140239 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.140206 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-tmp-dir\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:36:08.140239 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.140235 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-tls-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:36:08.140406 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.140247 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-home\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:36:08.140406 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.140258 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-dshm\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:36:08.140406 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.140269 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqmps\" (UniqueName: \"kubernetes.io/projected/4bb89e7b-26cc-47a4-85bd-c38a15e3c150-kube-api-access-dqmps\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:36:08.357025 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.356988 2572 generic.go:356] "Generic (PLEG): container finished" podID="4bb89e7b-26cc-47a4-85bd-c38a15e3c150" containerID="9046e14cdf77fc5caa7f97d553d25db5f153806da8b1bbeeedba4705f57fcef4" exitCode=0 Aug 04 18:36:08.357025 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.357026 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" Aug 04 18:36:08.357223 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.357055 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" event={"ID":"4bb89e7b-26cc-47a4-85bd-c38a15e3c150","Type":"ContainerDied","Data":"9046e14cdf77fc5caa7f97d553d25db5f153806da8b1bbeeedba4705f57fcef4"} Aug 04 18:36:08.357223 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.357093 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf" event={"ID":"4bb89e7b-26cc-47a4-85bd-c38a15e3c150","Type":"ContainerDied","Data":"50f96e79791cae255d75b0e4552669fbc086c114c8013a696f5fd308778e00f1"} Aug 04 18:36:08.357223 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.357114 2572 scope.go:117] "RemoveContainer" containerID="9046e14cdf77fc5caa7f97d553d25db5f153806da8b1bbeeedba4705f57fcef4" Aug 04 18:36:08.366102 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.366085 2572 scope.go:117] "RemoveContainer" containerID="9046e14cdf77fc5caa7f97d553d25db5f153806da8b1bbeeedba4705f57fcef4" Aug 04 18:36:08.366369 ip-10-0-137-173 kubenswrapper[2572]: E0804 18:36:08.366352 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9046e14cdf77fc5caa7f97d553d25db5f153806da8b1bbeeedba4705f57fcef4\": container with ID starting with 9046e14cdf77fc5caa7f97d553d25db5f153806da8b1bbeeedba4705f57fcef4 not found: ID does not exist" containerID="9046e14cdf77fc5caa7f97d553d25db5f153806da8b1bbeeedba4705f57fcef4" Aug 04 18:36:08.366442 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.366376 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9046e14cdf77fc5caa7f97d553d25db5f153806da8b1bbeeedba4705f57fcef4"} err="failed to get container status \"9046e14cdf77fc5caa7f97d553d25db5f153806da8b1bbeeedba4705f57fcef4\": rpc error: code = NotFound desc = could not find container \"9046e14cdf77fc5caa7f97d553d25db5f153806da8b1bbeeedba4705f57fcef4\": container with ID starting with 9046e14cdf77fc5caa7f97d553d25db5f153806da8b1bbeeedba4705f57fcef4 not found: ID does not exist" Aug 04 18:36:08.379069 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.379049 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf"] Aug 04 18:36:08.383540 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.383522 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c70eb3/test-model-c70eb3-kserve-6f6855d9cc-dh4tf"] Aug 04 18:36:08.792275 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.792190 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" podUID="68c82a75-6938-4e79-9be1-b734b6840877" containerName="main" containerID="cri-o://b2d98b18c1b5fd7095e6178c438f5fbfcbe022cd2a71e021c6608025ded70627" gracePeriod=45 Aug 04 18:36:08.821163 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:08.821125 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bb89e7b-26cc-47a4-85bd-c38a15e3c150" path="/var/lib/kubelet/pods/4bb89e7b-26cc-47a4-85bd-c38a15e3c150/volumes" Aug 04 18:36:09.038085 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.038059 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:36:09.148957 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.148917 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c82a75-6938-4e79-9be1-b734b6840877-tmp-dir\") pod \"68c82a75-6938-4e79-9be1-b734b6840877\" (UID: \"68c82a75-6938-4e79-9be1-b734b6840877\") " Aug 04 18:36:09.148957 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.148964 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c82a75-6938-4e79-9be1-b734b6840877-dshm\") pod \"68c82a75-6938-4e79-9be1-b734b6840877\" (UID: \"68c82a75-6938-4e79-9be1-b734b6840877\") " Aug 04 18:36:09.149215 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.149007 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68c82a75-6938-4e79-9be1-b734b6840877-home\") pod \"68c82a75-6938-4e79-9be1-b734b6840877\" (UID: \"68c82a75-6938-4e79-9be1-b734b6840877\") " Aug 04 18:36:09.149215 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.149047 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcpd2\" (UniqueName: \"kubernetes.io/projected/68c82a75-6938-4e79-9be1-b734b6840877-kube-api-access-xcpd2\") pod \"68c82a75-6938-4e79-9be1-b734b6840877\" (UID: \"68c82a75-6938-4e79-9be1-b734b6840877\") " Aug 04 18:36:09.149215 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.149080 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c82a75-6938-4e79-9be1-b734b6840877-model-cache\") pod \"68c82a75-6938-4e79-9be1-b734b6840877\" (UID: \"68c82a75-6938-4e79-9be1-b734b6840877\") " Aug 04 18:36:09.149215 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.149129 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68c82a75-6938-4e79-9be1-b734b6840877-tls-certs\") pod \"68c82a75-6938-4e79-9be1-b734b6840877\" (UID: \"68c82a75-6938-4e79-9be1-b734b6840877\") " Aug 04 18:36:09.149427 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.149266 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c82a75-6938-4e79-9be1-b734b6840877-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "68c82a75-6938-4e79-9be1-b734b6840877" (UID: "68c82a75-6938-4e79-9be1-b734b6840877"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:36:09.149427 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.149279 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c82a75-6938-4e79-9be1-b734b6840877-home" (OuterVolumeSpecName: "home") pod "68c82a75-6938-4e79-9be1-b734b6840877" (UID: "68c82a75-6938-4e79-9be1-b734b6840877"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:36:09.149427 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.149367 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c82a75-6938-4e79-9be1-b734b6840877-model-cache" (OuterVolumeSpecName: "model-cache") pod "68c82a75-6938-4e79-9be1-b734b6840877" (UID: "68c82a75-6938-4e79-9be1-b734b6840877"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:36:09.149554 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.149443 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68c82a75-6938-4e79-9be1-b734b6840877-tmp-dir\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:36:09.149554 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.149464 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68c82a75-6938-4e79-9be1-b734b6840877-home\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:36:09.151426 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.151393 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c82a75-6938-4e79-9be1-b734b6840877-dshm" (OuterVolumeSpecName: "dshm") pod "68c82a75-6938-4e79-9be1-b734b6840877" (UID: "68c82a75-6938-4e79-9be1-b734b6840877"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:36:09.151543 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.151446 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c82a75-6938-4e79-9be1-b734b6840877-kube-api-access-xcpd2" (OuterVolumeSpecName: "kube-api-access-xcpd2") pod "68c82a75-6938-4e79-9be1-b734b6840877" (UID: "68c82a75-6938-4e79-9be1-b734b6840877"). InnerVolumeSpecName "kube-api-access-xcpd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:36:09.151543 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.151457 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68c82a75-6938-4e79-9be1-b734b6840877-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "68c82a75-6938-4e79-9be1-b734b6840877" (UID: "68c82a75-6938-4e79-9be1-b734b6840877"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:36:09.250619 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.250584 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68c82a75-6938-4e79-9be1-b734b6840877-tls-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:36:09.250619 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.250612 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68c82a75-6938-4e79-9be1-b734b6840877-dshm\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:36:09.250619 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.250621 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcpd2\" (UniqueName: \"kubernetes.io/projected/68c82a75-6938-4e79-9be1-b734b6840877-kube-api-access-xcpd2\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:36:09.250858 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.250632 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68c82a75-6938-4e79-9be1-b734b6840877-model-cache\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:36:09.364316 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.364281 2572 generic.go:356] "Generic (PLEG): container finished" podID="68c82a75-6938-4e79-9be1-b734b6840877" containerID="b2d98b18c1b5fd7095e6178c438f5fbfcbe022cd2a71e021c6608025ded70627" exitCode=0 Aug 04 18:36:09.364504 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.364330 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" event={"ID":"68c82a75-6938-4e79-9be1-b734b6840877","Type":"ContainerDied","Data":"b2d98b18c1b5fd7095e6178c438f5fbfcbe022cd2a71e021c6608025ded70627"} Aug 04 18:36:09.364504 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.364332 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" Aug 04 18:36:09.364504 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.364360 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2" event={"ID":"68c82a75-6938-4e79-9be1-b734b6840877","Type":"ContainerDied","Data":"3b4c70623900f91daa2ecbaa3361c9361ae4e3d00bc386df0ea83faf819ee671"} Aug 04 18:36:09.364504 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.364379 2572 scope.go:117] "RemoveContainer" containerID="b2d98b18c1b5fd7095e6178c438f5fbfcbe022cd2a71e021c6608025ded70627" Aug 04 18:36:09.373718 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.373701 2572 scope.go:117] "RemoveContainer" containerID="b2d98b18c1b5fd7095e6178c438f5fbfcbe022cd2a71e021c6608025ded70627" Aug 04 18:36:09.373983 ip-10-0-137-173 kubenswrapper[2572]: E0804 18:36:09.373964 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d98b18c1b5fd7095e6178c438f5fbfcbe022cd2a71e021c6608025ded70627\": container with ID starting with b2d98b18c1b5fd7095e6178c438f5fbfcbe022cd2a71e021c6608025ded70627 not found: ID does not exist" containerID="b2d98b18c1b5fd7095e6178c438f5fbfcbe022cd2a71e021c6608025ded70627" Aug 04 18:36:09.374041 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.373991 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d98b18c1b5fd7095e6178c438f5fbfcbe022cd2a71e021c6608025ded70627"} err="failed to get container status \"b2d98b18c1b5fd7095e6178c438f5fbfcbe022cd2a71e021c6608025ded70627\": rpc error: code = NotFound desc = could not find container \"b2d98b18c1b5fd7095e6178c438f5fbfcbe022cd2a71e021c6608025ded70627\": container with ID starting with b2d98b18c1b5fd7095e6178c438f5fbfcbe022cd2a71e021c6608025ded70627 not found: ID does not exist" Aug 04 18:36:09.385740 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.385713 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2"] Aug 04 18:36:09.390982 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:09.390962 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7f7c0a/test-model-7f7c0a-kserve-656b6477f7-sshx2"] Aug 04 18:36:10.821939 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:10.821910 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68c82a75-6938-4e79-9be1-b734b6840877" path="/var/lib/kubelet/pods/68c82a75-6938-4e79-9be1-b734b6840877/volumes" Aug 04 18:36:12.590805 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:12.590770 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:12.693128 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:12.693085 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:12.911682 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:12.911582 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:13.137165 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:13.137131 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:13.292067 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:13.291991 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f7c0a-8c2ls"] Aug 04 18:36:13.292353 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:13.292340 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bb89e7b-26cc-47a4-85bd-c38a15e3c150" containerName="main" Aug 04 18:36:13.292399 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:13.292355 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bb89e7b-26cc-47a4-85bd-c38a15e3c150" containerName="main" Aug 04 18:36:13.292399 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:13.292377 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68c82a75-6938-4e79-9be1-b734b6840877" containerName="main" Aug 04 18:36:13.292399 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:13.292382 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="68c82a75-6938-4e79-9be1-b734b6840877" containerName="main" Aug 04 18:36:13.292495 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:13.292447 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bb89e7b-26cc-47a4-85bd-c38a15e3c150" containerName="main" Aug 04 18:36:13.292495 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:13.292457 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="68c82a75-6938-4e79-9be1-b734b6840877" containerName="main" Aug 04 18:36:13.296900 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:13.296886 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f7c0a-8c2ls" Aug 04 18:36:13.302391 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:13.302364 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pnbq7\"" Aug 04 18:36:13.302935 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:13.302916 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 18:36:13.313823 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:13.313789 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f7c0a-8c2ls"] Aug 04 18:36:13.387074 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:13.387039 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e5b3d6-a23d-4bcc-aefd-977d72f5a865-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7f7c0a-8c2ls\" (UID: \"62e5b3d6-a23d-4bcc-aefd-977d72f5a865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f7c0a-8c2ls" Aug 04 18:36:13.487846 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:13.487814 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e5b3d6-a23d-4bcc-aefd-977d72f5a865-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7f7c0a-8c2ls\" (UID: \"62e5b3d6-a23d-4bcc-aefd-977d72f5a865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f7c0a-8c2ls" Aug 04 18:36:13.488433 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:13.488413 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e5b3d6-a23d-4bcc-aefd-977d72f5a865-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7f7c0a-8c2ls\" (UID: \"62e5b3d6-a23d-4bcc-aefd-977d72f5a865\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f7c0a-8c2ls" Aug 04 18:36:13.607510 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:13.607469 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f7c0a-8c2ls" Aug 04 18:36:13.752616 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:13.752583 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f7c0a-8c2ls"] Aug 04 18:36:13.771902 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:36:13.771868 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e5b3d6_a23d_4bcc_aefd_977d72f5a865.slice/crio-7436e3c3ef97bbba71d65dbc41dcb6228895e3b3997d440e73745b8b60ee655d WatchSource:0}: Error finding container 7436e3c3ef97bbba71d65dbc41dcb6228895e3b3997d440e73745b8b60ee655d: Status 404 returned error can't find the container with id 7436e3c3ef97bbba71d65dbc41dcb6228895e3b3997d440e73745b8b60ee655d Aug 04 18:36:14.389784 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:14.389744 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f7c0a-8c2ls" event={"ID":"62e5b3d6-a23d-4bcc-aefd-977d72f5a865","Type":"ContainerStarted","Data":"f1ac4b9ec830bc568f7a82689e374627516d23964a1e0806ea0c4f723aad1e2f"} Aug 04 18:36:14.389945 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:14.389812 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f7c0a-8c2ls" event={"ID":"62e5b3d6-a23d-4bcc-aefd-977d72f5a865","Type":"ContainerStarted","Data":"7436e3c3ef97bbba71d65dbc41dcb6228895e3b3997d440e73745b8b60ee655d"} Aug 04 18:36:14.418000 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:14.417950 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f7c0a-8c2ls" podStartSLOduration=1.417933802 podStartE2EDuration="1.417933802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:36:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:36:14.41730464 +0000 UTC m=+2630.107995701" watchObservedRunningTime="2026-08-04 18:36:14.417933802 +0000 UTC m=+2630.108624852" Aug 04 18:36:15.395780 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:15.395745 2572 generic.go:356] "Generic (PLEG): container finished" podID="62e5b3d6-a23d-4bcc-aefd-977d72f5a865" containerID="f1ac4b9ec830bc568f7a82689e374627516d23964a1e0806ea0c4f723aad1e2f" exitCode=0 Aug 04 18:36:15.396207 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:15.395824 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f7c0a-8c2ls" event={"ID":"62e5b3d6-a23d-4bcc-aefd-977d72f5a865","Type":"ContainerDied","Data":"f1ac4b9ec830bc568f7a82689e374627516d23964a1e0806ea0c4f723aad1e2f"} Aug 04 18:36:16.530109 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:16.530085 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f7c0a-8c2ls" Aug 04 18:36:16.612498 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:16.612465 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e5b3d6-a23d-4bcc-aefd-977d72f5a865-maas-api-service-ca\") pod \"62e5b3d6-a23d-4bcc-aefd-977d72f5a865\" (UID: \"62e5b3d6-a23d-4bcc-aefd-977d72f5a865\") " Aug 04 18:36:16.612881 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:16.612862 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e5b3d6-a23d-4bcc-aefd-977d72f5a865-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "62e5b3d6-a23d-4bcc-aefd-977d72f5a865" (UID: "62e5b3d6-a23d-4bcc-aefd-977d72f5a865"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:36:16.713866 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:16.713781 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e5b3d6-a23d-4bcc-aefd-977d72f5a865-maas-api-service-ca\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:36:17.407457 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:17.407428 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f7c0a-8c2ls" Aug 04 18:36:17.407457 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:17.407432 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f7c0a-8c2ls" event={"ID":"62e5b3d6-a23d-4bcc-aefd-977d72f5a865","Type":"ContainerDied","Data":"7436e3c3ef97bbba71d65dbc41dcb6228895e3b3997d440e73745b8b60ee655d"} Aug 04 18:36:17.407686 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:17.407484 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7436e3c3ef97bbba71d65dbc41dcb6228895e3b3997d440e73745b8b60ee655d" Aug 04 18:36:18.841509 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:18.841474 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:18.943319 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:18.943285 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:19.040525 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:19.040494 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:19.070105 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:19.070063 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f7c0a-8c2ls"] Aug 04 18:36:19.074942 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:19.074914 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7f7c0a-8c2ls"] Aug 04 18:36:19.102042 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:19.101978 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:19.458060 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:19.457971 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:20.821991 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:20.821955 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62e5b3d6-a23d-4bcc-aefd-977d72f5a865" path="/var/lib/kubelet/pods/62e5b3d6-a23d-4bcc-aefd-977d72f5a865/volumes" Aug 04 18:36:25.028300 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:25.028271 2572 scope.go:117] "RemoveContainer" containerID="a731b51b9d90b91c532578edb6495f9302cb819882f6c233aaefc6333ae26a45" Aug 04 18:36:26.324204 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:26.324174 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:26.424853 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:26.424823 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:26.536899 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:26.536869 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s"] Aug 04 18:36:26.537095 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:26.537070 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" podUID="46544f7d-0eca-4aa2-83ab-86aea841ecf6" containerName="istio-proxy" containerID="cri-o://6628696e25c261abcab80a62ae5a5f7b4e6d16189747b38aba4bdc182e1275c8" gracePeriod=30 Aug 04 18:36:26.549940 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:26.549918 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:26.768391 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:26.768299 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:27.097882 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:27.097848 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:27.128822 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:27.128789 2572 patch_prober.go:28] interesting pod/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 18:36:27.129003 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:27.128864 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" podUID="46544f7d-0eca-4aa2-83ab-86aea841ecf6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 18:36:27.432926 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:27.432843 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:27.551109 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:27.551077 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:30.234770 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:30.234734 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:30.326061 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:30.326030 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:30.428568 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:30.428539 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:31.426208 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.426176 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:31.784283 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.784251 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:31.785617 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.785597 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:36:31.948162 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.948131 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istiod-ca-cert\") pod \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " Aug 04 18:36:31.948329 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.948174 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istio-podinfo\") pod \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " Aug 04 18:36:31.948329 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.948198 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-workload-socket\") pod \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " Aug 04 18:36:31.948329 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.948224 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-credential-socket\") pod \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " Aug 04 18:36:31.948329 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.948245 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istio-data\") pod \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " Aug 04 18:36:31.948329 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.948267 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istio-token\") pod \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " Aug 04 18:36:31.948329 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.948330 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istio-envoy\") pod \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " Aug 04 18:36:31.948652 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.948361 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88drt\" (UniqueName: \"kubernetes.io/projected/46544f7d-0eca-4aa2-83ab-86aea841ecf6-kube-api-access-88drt\") pod \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " Aug 04 18:36:31.948652 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.948385 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-workload-certs\") pod \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\" (UID: \"46544f7d-0eca-4aa2-83ab-86aea841ecf6\") " Aug 04 18:36:31.948652 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.948560 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "46544f7d-0eca-4aa2-83ab-86aea841ecf6" (UID: "46544f7d-0eca-4aa2-83ab-86aea841ecf6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:36:31.948652 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.948570 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "46544f7d-0eca-4aa2-83ab-86aea841ecf6" (UID: "46544f7d-0eca-4aa2-83ab-86aea841ecf6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:36:31.948652 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.948590 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istio-data" (OuterVolumeSpecName: "istio-data") pod "46544f7d-0eca-4aa2-83ab-86aea841ecf6" (UID: "46544f7d-0eca-4aa2-83ab-86aea841ecf6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:36:31.948940 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.948690 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "46544f7d-0eca-4aa2-83ab-86aea841ecf6" (UID: "46544f7d-0eca-4aa2-83ab-86aea841ecf6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:36:31.948940 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.948868 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "46544f7d-0eca-4aa2-83ab-86aea841ecf6" (UID: "46544f7d-0eca-4aa2-83ab-86aea841ecf6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:36:31.949059 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.949037 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-workload-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:36:31.949116 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.949058 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istiod-ca-cert\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:36:31.949116 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.949073 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-workload-socket\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:36:31.949116 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.949086 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-credential-socket\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:36:31.949116 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.949097 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istio-data\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:36:31.950771 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.950742 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "46544f7d-0eca-4aa2-83ab-86aea841ecf6" (UID: "46544f7d-0eca-4aa2-83ab-86aea841ecf6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:36:31.950924 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.950833 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istio-token" (OuterVolumeSpecName: "istio-token") pod "46544f7d-0eca-4aa2-83ab-86aea841ecf6" (UID: "46544f7d-0eca-4aa2-83ab-86aea841ecf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:36:31.951206 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.951182 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "46544f7d-0eca-4aa2-83ab-86aea841ecf6" (UID: "46544f7d-0eca-4aa2-83ab-86aea841ecf6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:36:31.951206 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:31.951186 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46544f7d-0eca-4aa2-83ab-86aea841ecf6-kube-api-access-88drt" (OuterVolumeSpecName: "kube-api-access-88drt") pod "46544f7d-0eca-4aa2-83ab-86aea841ecf6" (UID: "46544f7d-0eca-4aa2-83ab-86aea841ecf6"). InnerVolumeSpecName "kube-api-access-88drt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:36:32.017961 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:32.017874 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:32.050516 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:32.050495 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istio-podinfo\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:36:32.050516 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:32.050514 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istio-token\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:36:32.050695 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:32.050525 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46544f7d-0eca-4aa2-83ab-86aea841ecf6-istio-envoy\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:36:32.050695 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:32.050533 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88drt\" (UniqueName: \"kubernetes.io/projected/46544f7d-0eca-4aa2-83ab-86aea841ecf6-kube-api-access-88drt\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:36:32.126633 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:32.126586 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:32.479327 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:32.479291 2572 generic.go:356] "Generic (PLEG): container finished" podID="46544f7d-0eca-4aa2-83ab-86aea841ecf6" containerID="6628696e25c261abcab80a62ae5a5f7b4e6d16189747b38aba4bdc182e1275c8" exitCode=0 Aug 04 18:36:32.479777 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:32.479336 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" Aug 04 18:36:32.479777 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:32.479379 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" event={"ID":"46544f7d-0eca-4aa2-83ab-86aea841ecf6","Type":"ContainerDied","Data":"6628696e25c261abcab80a62ae5a5f7b4e6d16189747b38aba4bdc182e1275c8"} Aug 04 18:36:32.479777 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:32.479433 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s" event={"ID":"46544f7d-0eca-4aa2-83ab-86aea841ecf6","Type":"ContainerDied","Data":"b2ccf9af1c8e7b2767a082b12869e95eeca118d4f31c900fb28ecdb9cdd30b33"} Aug 04 18:36:32.479777 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:32.479456 2572 scope.go:117] "RemoveContainer" containerID="6628696e25c261abcab80a62ae5a5f7b4e6d16189747b38aba4bdc182e1275c8" Aug 04 18:36:32.488458 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:32.488442 2572 scope.go:117] "RemoveContainer" containerID="258c6b88b1ae43de4bd5e44aaecdc77ceb4d79d8cf11cd400a81a76bb8db6543" Aug 04 18:36:32.496105 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:32.496085 2572 scope.go:117] "RemoveContainer" containerID="6628696e25c261abcab80a62ae5a5f7b4e6d16189747b38aba4bdc182e1275c8" Aug 04 18:36:32.496343 ip-10-0-137-173 kubenswrapper[2572]: E0804 18:36:32.496324 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6628696e25c261abcab80a62ae5a5f7b4e6d16189747b38aba4bdc182e1275c8\": container with ID starting with 6628696e25c261abcab80a62ae5a5f7b4e6d16189747b38aba4bdc182e1275c8 not found: ID does not exist" containerID="6628696e25c261abcab80a62ae5a5f7b4e6d16189747b38aba4bdc182e1275c8" Aug 04 18:36:32.496414 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:32.496357 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6628696e25c261abcab80a62ae5a5f7b4e6d16189747b38aba4bdc182e1275c8"} err="failed to get container status \"6628696e25c261abcab80a62ae5a5f7b4e6d16189747b38aba4bdc182e1275c8\": rpc error: code = NotFound desc = could not find container \"6628696e25c261abcab80a62ae5a5f7b4e6d16189747b38aba4bdc182e1275c8\": container with ID starting with 6628696e25c261abcab80a62ae5a5f7b4e6d16189747b38aba4bdc182e1275c8 not found: ID does not exist" Aug 04 18:36:32.496414 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:32.496379 2572 scope.go:117] "RemoveContainer" containerID="258c6b88b1ae43de4bd5e44aaecdc77ceb4d79d8cf11cd400a81a76bb8db6543" Aug 04 18:36:32.496589 ip-10-0-137-173 kubenswrapper[2572]: E0804 18:36:32.496574 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"258c6b88b1ae43de4bd5e44aaecdc77ceb4d79d8cf11cd400a81a76bb8db6543\": container with ID starting with 258c6b88b1ae43de4bd5e44aaecdc77ceb4d79d8cf11cd400a81a76bb8db6543 not found: ID does not exist" containerID="258c6b88b1ae43de4bd5e44aaecdc77ceb4d79d8cf11cd400a81a76bb8db6543" Aug 04 18:36:32.496640 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:32.496596 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"258c6b88b1ae43de4bd5e44aaecdc77ceb4d79d8cf11cd400a81a76bb8db6543"} err="failed to get container status \"258c6b88b1ae43de4bd5e44aaecdc77ceb4d79d8cf11cd400a81a76bb8db6543\": rpc error: code = NotFound desc = could not find container \"258c6b88b1ae43de4bd5e44aaecdc77ceb4d79d8cf11cd400a81a76bb8db6543\": container with ID starting with 258c6b88b1ae43de4bd5e44aaecdc77ceb4d79d8cf11cd400a81a76bb8db6543 not found: ID does not exist" Aug 04 18:36:32.508508 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:32.508485 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s"] Aug 04 18:36:32.510747 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:32.510728 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7f7c0a-openshift-default-69c44bcdbd-lpt9s"] Aug 04 18:36:32.821919 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:32.821888 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46544f7d-0eca-4aa2-83ab-86aea841ecf6" path="/var/lib/kubelet/pods/46544f7d-0eca-4aa2-83ab-86aea841ecf6/volumes" Aug 04 18:36:55.183180 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:55.183143 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:55.664818 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:55.664786 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:56.035258 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.035178 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz"] Aug 04 18:36:56.035534 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.035521 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46544f7d-0eca-4aa2-83ab-86aea841ecf6" containerName="istio-proxy" Aug 04 18:36:56.035580 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.035535 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="46544f7d-0eca-4aa2-83ab-86aea841ecf6" containerName="istio-proxy" Aug 04 18:36:56.035580 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.035549 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62e5b3d6-a23d-4bcc-aefd-977d72f5a865" containerName="revoke-keys" Aug 04 18:36:56.035580 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.035555 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="62e5b3d6-a23d-4bcc-aefd-977d72f5a865" containerName="revoke-keys" Aug 04 18:36:56.035691 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.035604 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="62e5b3d6-a23d-4bcc-aefd-977d72f5a865" containerName="revoke-keys" Aug 04 18:36:56.035691 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.035611 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="46544f7d-0eca-4aa2-83ab-86aea841ecf6" containerName="istio-proxy" Aug 04 18:36:56.035691 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.035619 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="46544f7d-0eca-4aa2-83ab-86aea841ecf6" containerName="istio-proxy" Aug 04 18:36:56.039879 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.039859 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.046137 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.046115 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a77bad-openshift-default-dockercfg-8qxn4\"" Aug 04 18:36:56.048174 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.048142 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:56.066170 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.066147 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz"] Aug 04 18:36:56.125133 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.125097 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-workload-socket\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.125307 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.125144 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8z7h\" (UniqueName: \"kubernetes.io/projected/94683587-4d29-4f55-b403-09fedab86a7c-kube-api-access-m8z7h\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.125307 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.125167 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-istio-data\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.125307 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.125195 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-credential-socket\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.125307 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.125216 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94683587-4d29-4f55-b403-09fedab86a7c-istio-token\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.125307 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.125231 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94683587-4d29-4f55-b403-09fedab86a7c-istiod-ca-cert\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.125307 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.125260 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-istio-envoy\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.125307 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.125297 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-workload-certs\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.125540 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.125320 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94683587-4d29-4f55-b403-09fedab86a7c-istio-podinfo\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.226135 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.226101 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-workload-socket\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.226566 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.226150 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8z7h\" (UniqueName: \"kubernetes.io/projected/94683587-4d29-4f55-b403-09fedab86a7c-kube-api-access-m8z7h\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.226566 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.226172 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-istio-data\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.226566 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.226198 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-credential-socket\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.226566 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.226212 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94683587-4d29-4f55-b403-09fedab86a7c-istio-token\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.226566 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.226229 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94683587-4d29-4f55-b403-09fedab86a7c-istiod-ca-cert\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.226566 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.226247 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-istio-envoy\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.226566 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.226284 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-workload-certs\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.226566 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.226321 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94683587-4d29-4f55-b403-09fedab86a7c-istio-podinfo\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.227011 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.226581 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-workload-socket\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.227011 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.226612 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-credential-socket\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.227011 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.226725 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-istio-data\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.227011 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.226845 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-workload-certs\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.227011 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.226995 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94683587-4d29-4f55-b403-09fedab86a7c-istiod-ca-cert\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.228853 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.228835 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-istio-envoy\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.228937 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.228893 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94683587-4d29-4f55-b403-09fedab86a7c-istio-podinfo\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.234050 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.234027 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:56.244775 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.244753 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8z7h\" (UniqueName: \"kubernetes.io/projected/94683587-4d29-4f55-b403-09fedab86a7c-kube-api-access-m8z7h\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.245909 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.245887 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94683587-4d29-4f55-b403-09fedab86a7c-istio-token\") pod \"e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.350870 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.350834 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:56.501532 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.501508 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz"] Aug 04 18:36:56.504073 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:36:56.504039 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94683587_4d29_4f55_b403_09fedab86a7c.slice/crio-e005e991628d64376b4f15c05930cf6c8e2d5f37861de64ff82d9a47fcd0df80 WatchSource:0}: Error finding container e005e991628d64376b4f15c05930cf6c8e2d5f37861de64ff82d9a47fcd0df80: Status 404 returned error can't find the container with id e005e991628d64376b4f15c05930cf6c8e2d5f37861de64ff82d9a47fcd0df80 Aug 04 18:36:56.506562 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.506529 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:36:56.506680 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.506606 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:36:56.506680 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.506647 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:36:56.600977 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.600886 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" event={"ID":"94683587-4d29-4f55-b403-09fedab86a7c","Type":"ContainerStarted","Data":"23d72a728cb0c4e3fdae30c04dece297a953c779f5254dd7b3e56d0c8f90bc77"} Aug 04 18:36:56.600977 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.600946 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" event={"ID":"94683587-4d29-4f55-b403-09fedab86a7c","Type":"ContainerStarted","Data":"e005e991628d64376b4f15c05930cf6c8e2d5f37861de64ff82d9a47fcd0df80"} Aug 04 18:36:56.647565 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:56.647518 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" podStartSLOduration=1.647503075 podStartE2EDuration="1.647503075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:36:56.643067333 +0000 UTC m=+2672.333758384" watchObservedRunningTime="2026-08-04 18:36:56.647503075 +0000 UTC m=+2672.338194125" Aug 04 18:36:57.351648 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:57.351616 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:57.356535 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:57.356511 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:57.605782 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:57.605698 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:57.606610 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:57.606593 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:36:58.662626 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:58.662595 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:58.756912 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:58.756877 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:36:59.827220 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:36:59.827184 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:37:00.053785 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:00.053750 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:37:00.446089 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:00.446051 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:37:00.544379 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:00.544350 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:37:12.616653 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:12.616616 2572 patch_prober.go:28] interesting pod/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": read tcp 10.134.0.2:43960->10.134.0.38:15021: read: connection reset by peer" start-of-body= Aug 04 18:37:12.617066 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:12.616705 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" podUID="94683587-4d29-4f55-b403-09fedab86a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": read tcp 10.134.0.2:43960->10.134.0.38:15021: read: connection reset by peer" Aug 04 18:37:12.674307 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:12.674275 2572 generic.go:356] "Generic (PLEG): container finished" podID="94683587-4d29-4f55-b403-09fedab86a7c" containerID="23d72a728cb0c4e3fdae30c04dece297a953c779f5254dd7b3e56d0c8f90bc77" exitCode=0 Aug 04 18:37:12.674431 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:12.674335 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" event={"ID":"94683587-4d29-4f55-b403-09fedab86a7c","Type":"ContainerDied","Data":"23d72a728cb0c4e3fdae30c04dece297a953c779f5254dd7b3e56d0c8f90bc77"} Aug 04 18:37:12.674773 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:12.674755 2572 scope.go:117] "RemoveContainer" containerID="23d72a728cb0c4e3fdae30c04dece297a953c779f5254dd7b3e56d0c8f90bc77" Aug 04 18:37:12.675768 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:12.675738 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:37:12.675860 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:12.675803 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:37:12.675860 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:12.675832 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:37:13.685003 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:13.684960 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" event={"ID":"94683587-4d29-4f55-b403-09fedab86a7c","Type":"ContainerStarted","Data":"6f16f8d2867017295a5d19151846dcc0ca920f202f072bf888e86c2f989ca45a"} Aug 04 18:37:14.351495 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:14.351456 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:37:14.351709 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:14.351687 2572 patch_prober.go:28] interesting pod/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Aug 04 18:37:14.351760 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:14.351734 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" podUID="94683587-4d29-4f55-b403-09fedab86a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Aug 04 18:37:15.351937 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:15.351862 2572 patch_prober.go:28] interesting pod/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Aug 04 18:37:15.351937 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:15.351916 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" podUID="94683587-4d29-4f55-b403-09fedab86a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Aug 04 18:37:16.351404 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:16.351367 2572 patch_prober.go:28] interesting pod/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Aug 04 18:37:16.351584 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:16.351423 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" podUID="94683587-4d29-4f55-b403-09fedab86a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Aug 04 18:37:17.351287 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:17.351251 2572 patch_prober.go:28] interesting pod/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Aug 04 18:37:17.351691 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:17.351307 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" podUID="94683587-4d29-4f55-b403-09fedab86a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Aug 04 18:37:18.351355 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:18.351320 2572 patch_prober.go:28] interesting pod/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Aug 04 18:37:18.351728 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:18.351370 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" podUID="94683587-4d29-4f55-b403-09fedab86a7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Aug 04 18:37:19.355323 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:19.355296 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:37:19.355775 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:19.355554 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:37:19.356274 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:19.356259 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:37:25.055138 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:25.055106 2572 scope.go:117] "RemoveContainer" containerID="c4c6a1734dfdafcaa4e357aa8af287bd593c62a757ebf33cbc69be016d6e243a" Aug 04 18:37:28.769886 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:28.769851 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:37:31.677445 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:31.677413 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a08d36-j68c5"] Aug 04 18:37:31.677828 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:31.677792 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46544f7d-0eca-4aa2-83ab-86aea841ecf6" containerName="istio-proxy" Aug 04 18:37:31.677828 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:31.677806 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="46544f7d-0eca-4aa2-83ab-86aea841ecf6" containerName="istio-proxy" Aug 04 18:37:31.679783 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:31.679768 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a08d36-j68c5" Aug 04 18:37:31.682623 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:31.682605 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 18:37:31.682752 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:31.682608 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pnbq7\"" Aug 04 18:37:31.688903 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:31.688875 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a08d36-j68c5"] Aug 04 18:37:31.834604 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:31.834571 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7be0d0ff-d084-461b-ac66-8582ddf60582-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a08d36-j68c5\" (UID: \"7be0d0ff-d084-461b-ac66-8582ddf60582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a08d36-j68c5" Aug 04 18:37:31.935213 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:31.935112 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7be0d0ff-d084-461b-ac66-8582ddf60582-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a08d36-j68c5\" (UID: \"7be0d0ff-d084-461b-ac66-8582ddf60582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a08d36-j68c5" Aug 04 18:37:31.935787 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:31.935769 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7be0d0ff-d084-461b-ac66-8582ddf60582-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a08d36-j68c5\" (UID: \"7be0d0ff-d084-461b-ac66-8582ddf60582\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a08d36-j68c5" Aug 04 18:37:31.990486 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:31.990458 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a08d36-j68c5" Aug 04 18:37:32.120352 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:32.120322 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a08d36-j68c5"] Aug 04 18:37:32.123166 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:37:32.123135 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be0d0ff_d084_461b_ac66_8582ddf60582.slice/crio-9ed233bb04c03b107ba5ff0aab957a68cea59126433d72dab68bcdf0050ea7d1 WatchSource:0}: Error finding container 9ed233bb04c03b107ba5ff0aab957a68cea59126433d72dab68bcdf0050ea7d1: Status 404 returned error can't find the container with id 9ed233bb04c03b107ba5ff0aab957a68cea59126433d72dab68bcdf0050ea7d1 Aug 04 18:37:32.779921 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:32.779873 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a08d36-j68c5" event={"ID":"7be0d0ff-d084-461b-ac66-8582ddf60582","Type":"ContainerStarted","Data":"9e8afcc72d4669946046564fef3c83402a0d16292ad100ac7499b8f819327e1d"} Aug 04 18:37:32.780405 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:32.779977 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a08d36-j68c5" event={"ID":"7be0d0ff-d084-461b-ac66-8582ddf60582","Type":"ContainerStarted","Data":"9ed233bb04c03b107ba5ff0aab957a68cea59126433d72dab68bcdf0050ea7d1"} Aug 04 18:37:32.796739 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:32.796691 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a08d36-j68c5" podStartSLOduration=1.796652539 podStartE2EDuration="1.796652539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:37:32.796056461 +0000 UTC m=+2708.486747510" watchObservedRunningTime="2026-08-04 18:37:32.796652539 +0000 UTC m=+2708.487343591" Aug 04 18:37:33.789054 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:33.789023 2572 generic.go:356] "Generic (PLEG): container finished" podID="7be0d0ff-d084-461b-ac66-8582ddf60582" containerID="9e8afcc72d4669946046564fef3c83402a0d16292ad100ac7499b8f819327e1d" exitCode=0 Aug 04 18:37:33.789432 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:33.789109 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a08d36-j68c5" event={"ID":"7be0d0ff-d084-461b-ac66-8582ddf60582","Type":"ContainerDied","Data":"9e8afcc72d4669946046564fef3c83402a0d16292ad100ac7499b8f819327e1d"} Aug 04 18:37:34.919732 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:34.919709 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a08d36-j68c5" Aug 04 18:37:35.063843 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:35.063764 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7be0d0ff-d084-461b-ac66-8582ddf60582-maas-api-service-ca\") pod \"7be0d0ff-d084-461b-ac66-8582ddf60582\" (UID: \"7be0d0ff-d084-461b-ac66-8582ddf60582\") " Aug 04 18:37:35.064073 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:35.064050 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7be0d0ff-d084-461b-ac66-8582ddf60582-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7be0d0ff-d084-461b-ac66-8582ddf60582" (UID: "7be0d0ff-d084-461b-ac66-8582ddf60582"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:37:35.164841 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:35.164819 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7be0d0ff-d084-461b-ac66-8582ddf60582-maas-api-service-ca\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:37:35.799612 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:35.799578 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a08d36-j68c5" Aug 04 18:37:35.799612 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:35.799580 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a08d36-j68c5" event={"ID":"7be0d0ff-d084-461b-ac66-8582ddf60582","Type":"ContainerDied","Data":"9ed233bb04c03b107ba5ff0aab957a68cea59126433d72dab68bcdf0050ea7d1"} Aug 04 18:37:35.799911 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:35.799629 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ed233bb04c03b107ba5ff0aab957a68cea59126433d72dab68bcdf0050ea7d1" Aug 04 18:37:47.352376 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:47.352333 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:37:47.493150 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:47.493115 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:37:47.578292 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:47.578262 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:37:47.777844 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:47.777776 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:37:47.787410 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:47.787378 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a08d36-j68c5"] Aug 04 18:37:47.791215 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:47.791195 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a08d36-j68c5"] Aug 04 18:37:48.146717 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:48.146679 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:37:48.821875 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:48.821846 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7be0d0ff-d084-461b-ac66-8582ddf60582" path="/var/lib/kubelet/pods/7be0d0ff-d084-461b-ac66-8582ddf60582/volumes" Aug 04 18:37:54.394065 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:54.394032 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:37:54.455193 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:54.455154 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:37:54.644600 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:54.644515 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:37:54.842838 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:54.842803 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:37:54.970195 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:54.970117 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77bad-lvtr5"] Aug 04 18:37:54.970434 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:54.970422 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7be0d0ff-d084-461b-ac66-8582ddf60582" containerName="revoke-keys" Aug 04 18:37:54.970482 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:54.970436 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7be0d0ff-d084-461b-ac66-8582ddf60582" containerName="revoke-keys" Aug 04 18:37:54.970525 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:54.970492 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7be0d0ff-d084-461b-ac66-8582ddf60582" containerName="revoke-keys" Aug 04 18:37:54.972539 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:54.972523 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77bad-lvtr5" Aug 04 18:37:54.975061 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:54.975039 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pnbq7\"" Aug 04 18:37:54.975180 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:54.975139 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 18:37:54.981928 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:54.981905 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77bad-lvtr5"] Aug 04 18:37:55.034850 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:55.034820 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3da7c61-a052-4c0f-8e66-ba2aa28486cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a77bad-lvtr5\" (UID: \"b3da7c61-a052-4c0f-8e66-ba2aa28486cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77bad-lvtr5" Aug 04 18:37:55.136198 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:55.136168 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3da7c61-a052-4c0f-8e66-ba2aa28486cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a77bad-lvtr5\" (UID: \"b3da7c61-a052-4c0f-8e66-ba2aa28486cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77bad-lvtr5" Aug 04 18:37:55.136831 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:55.136810 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3da7c61-a052-4c0f-8e66-ba2aa28486cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a77bad-lvtr5\" (UID: \"b3da7c61-a052-4c0f-8e66-ba2aa28486cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77bad-lvtr5" Aug 04 18:37:55.284000 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:55.283935 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77bad-lvtr5" Aug 04 18:37:55.414085 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:55.413917 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77bad-lvtr5"] Aug 04 18:37:55.416283 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:37:55.416257 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3da7c61_a052_4c0f_8e66_ba2aa28486cb.slice/crio-a86ec8bed1e916e0a1e44e8c24a389e3a7a0ac9ce62ac8680a99a308e38b39b2 WatchSource:0}: Error finding container a86ec8bed1e916e0a1e44e8c24a389e3a7a0ac9ce62ac8680a99a308e38b39b2: Status 404 returned error can't find the container with id a86ec8bed1e916e0a1e44e8c24a389e3a7a0ac9ce62ac8680a99a308e38b39b2 Aug 04 18:37:55.896472 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:55.896391 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77bad-lvtr5" event={"ID":"b3da7c61-a052-4c0f-8e66-ba2aa28486cb","Type":"ContainerStarted","Data":"a39e4a4d763963a26b2d983105c388c65dac956ac98ed803fa0dacaeaa9d68b7"} Aug 04 18:37:55.896472 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:55.896454 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77bad-lvtr5" event={"ID":"b3da7c61-a052-4c0f-8e66-ba2aa28486cb","Type":"ContainerStarted","Data":"a86ec8bed1e916e0a1e44e8c24a389e3a7a0ac9ce62ac8680a99a308e38b39b2"} Aug 04 18:37:55.913017 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:55.912974 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77bad-lvtr5" podStartSLOduration=1.912957507 podStartE2EDuration="1.912957507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:37:55.912490238 +0000 UTC m=+2731.603181292" watchObservedRunningTime="2026-08-04 18:37:55.912957507 +0000 UTC m=+2731.603648557" Aug 04 18:37:56.902868 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:56.902832 2572 generic.go:356] "Generic (PLEG): container finished" podID="b3da7c61-a052-4c0f-8e66-ba2aa28486cb" containerID="a39e4a4d763963a26b2d983105c388c65dac956ac98ed803fa0dacaeaa9d68b7" exitCode=0 Aug 04 18:37:56.903230 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:56.902933 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77bad-lvtr5" event={"ID":"b3da7c61-a052-4c0f-8e66-ba2aa28486cb","Type":"ContainerDied","Data":"a39e4a4d763963a26b2d983105c388c65dac956ac98ed803fa0dacaeaa9d68b7"} Aug 04 18:37:58.034893 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:58.034872 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77bad-lvtr5" Aug 04 18:37:58.061033 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:58.061003 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3da7c61-a052-4c0f-8e66-ba2aa28486cb-maas-api-service-ca\") pod \"b3da7c61-a052-4c0f-8e66-ba2aa28486cb\" (UID: \"b3da7c61-a052-4c0f-8e66-ba2aa28486cb\") " Aug 04 18:37:58.061348 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:58.061326 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3da7c61-a052-4c0f-8e66-ba2aa28486cb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b3da7c61-a052-4c0f-8e66-ba2aa28486cb" (UID: "b3da7c61-a052-4c0f-8e66-ba2aa28486cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:37:58.161855 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:58.161787 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3da7c61-a052-4c0f-8e66-ba2aa28486cb-maas-api-service-ca\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:37:58.913736 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:58.913697 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77bad-lvtr5" event={"ID":"b3da7c61-a052-4c0f-8e66-ba2aa28486cb","Type":"ContainerDied","Data":"a86ec8bed1e916e0a1e44e8c24a389e3a7a0ac9ce62ac8680a99a308e38b39b2"} Aug 04 18:37:58.913892 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:58.913751 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a86ec8bed1e916e0a1e44e8c24a389e3a7a0ac9ce62ac8680a99a308e38b39b2" Aug 04 18:37:58.913892 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:37:58.913747 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77bad-lvtr5" Aug 04 18:38:10.501192 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:10.501160 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:10.571642 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:10.571596 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:10.633980 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:10.633941 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:10.694894 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:10.694866 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77bad-lvtr5"] Aug 04 18:38:10.697551 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:10.697525 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a77bad-lvtr5"] Aug 04 18:38:10.698365 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:10.698341 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:10.821379 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:10.821342 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3da7c61-a052-4c0f-8e66-ba2aa28486cb" path="/var/lib/kubelet/pods/b3da7c61-a052-4c0f-8e66-ba2aa28486cb/volumes" Aug 04 18:38:11.103085 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:11.103006 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:17.747906 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:17.747854 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:17.853026 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:17.852962 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:17.994630 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:17.994595 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz"] Aug 04 18:38:17.994890 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:17.994841 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" podUID="94683587-4d29-4f55-b403-09fedab86a7c" containerName="istio-proxy" containerID="cri-o://6f16f8d2867017295a5d19151846dcc0ca920f202f072bf888e86c2f989ca45a" gracePeriod=30 Aug 04 18:38:18.010103 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:18.010018 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:18.254642 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:18.254592 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:18.596165 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:18.596135 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:18.894456 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:18.894379 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9"] Aug 04 18:38:18.894810 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:18.894718 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3da7c61-a052-4c0f-8e66-ba2aa28486cb" containerName="revoke-keys" Aug 04 18:38:18.894810 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:18.894730 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3da7c61-a052-4c0f-8e66-ba2aa28486cb" containerName="revoke-keys" Aug 04 18:38:18.894810 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:18.894801 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3da7c61-a052-4c0f-8e66-ba2aa28486cb" containerName="revoke-keys" Aug 04 18:38:18.896828 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:18.896812 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:18.899532 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:18.899512 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-5c8685-openshift-default-dockercfg-lwzfq\"" Aug 04 18:38:18.911431 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:18.911408 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9"] Aug 04 18:38:18.925007 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:18.924981 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:19.016926 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.016894 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:19.054120 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.054080 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istio-data\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.054120 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.054128 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-workload-certs\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.054293 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.054197 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-credential-socket\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.054293 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.054220 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istio-token\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.054293 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.054276 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istio-podinfo\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.054406 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.054294 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istio-envoy\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.054406 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.054330 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istiod-ca-cert\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.054406 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.054350 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt2wk\" (UniqueName: \"kubernetes.io/projected/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-kube-api-access-rt2wk\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.054406 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.054376 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-workload-socket\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.155553 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.155470 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-workload-socket\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.155553 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.155513 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istio-data\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.155553 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.155534 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-workload-certs\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.155837 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.155568 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-credential-socket\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.155837 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.155589 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istio-token\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.155837 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.155612 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istio-podinfo\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.155837 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.155628 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istio-envoy\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.155837 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.155653 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istiod-ca-cert\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.155837 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.155691 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt2wk\" (UniqueName: \"kubernetes.io/projected/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-kube-api-access-rt2wk\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.156141 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.155941 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-workload-socket\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.156141 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.155967 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-credential-socket\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.156141 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.156124 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istio-data\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.156248 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.156155 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-workload-certs\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.156407 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.156383 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istiod-ca-cert\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.158321 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.158294 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istio-envoy\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.158465 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.158449 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istio-podinfo\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.165680 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.165643 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istio-token\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.166376 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.166355 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt2wk\" (UniqueName: \"kubernetes.io/projected/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-kube-api-access-rt2wk\") pod \"e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.207361 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.207335 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:19.339786 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.339752 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9"] Aug 04 18:38:19.342972 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:38:19.342946 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8a5bd9_0ad2_442b_94d9_8d4cc35e403d.slice/crio-5821c15f9ff3be2e9185b2cf07290b0546ae531da5ab12ba9197e4232fb91625 WatchSource:0}: Error finding container 5821c15f9ff3be2e9185b2cf07290b0546ae531da5ab12ba9197e4232fb91625: Status 404 returned error can't find the container with id 5821c15f9ff3be2e9185b2cf07290b0546ae531da5ab12ba9197e4232fb91625 Aug 04 18:38:19.345073 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.345032 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:38:19.345167 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.345106 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:38:19.345167 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.345135 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:38:19.357983 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.357958 2572 patch_prober.go:28] interesting pod/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 18:38:19.358064 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:19.358020 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" podUID="94683587-4d29-4f55-b403-09fedab86a7c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 18:38:20.011085 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:20.011045 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" event={"ID":"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d","Type":"ContainerStarted","Data":"356acf44522306aa3d55a8b6e95d07452f39aaf693dbaee13fb07f585d7726c1"} Aug 04 18:38:20.011510 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:20.011100 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" event={"ID":"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d","Type":"ContainerStarted","Data":"5821c15f9ff3be2e9185b2cf07290b0546ae531da5ab12ba9197e4232fb91625"} Aug 04 18:38:20.032678 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:20.032611 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" podStartSLOduration=2.032596835 podStartE2EDuration="2.032596835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:38:20.030839202 +0000 UTC m=+2755.721530262" watchObservedRunningTime="2026-08-04 18:38:20.032596835 +0000 UTC m=+2755.723287886" Aug 04 18:38:20.208425 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:20.208396 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:20.213573 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:20.213550 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:21.016173 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:21.016147 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:21.017006 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:21.016988 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:21.923908 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:21.923849 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:22.024008 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:22.023972 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:23.248841 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.248816 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:38:23.281703 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.281604 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:23.392840 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.392807 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94683587-4d29-4f55-b403-09fedab86a7c-istiod-ca-cert\") pod \"94683587-4d29-4f55-b403-09fedab86a7c\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " Aug 04 18:38:23.392840 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.392845 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94683587-4d29-4f55-b403-09fedab86a7c-istio-token\") pod \"94683587-4d29-4f55-b403-09fedab86a7c\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " Aug 04 18:38:23.393069 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.392862 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-istio-data\") pod \"94683587-4d29-4f55-b403-09fedab86a7c\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " Aug 04 18:38:23.393069 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.392904 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-workload-socket\") pod \"94683587-4d29-4f55-b403-09fedab86a7c\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " Aug 04 18:38:23.393069 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.392930 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-credential-socket\") pod \"94683587-4d29-4f55-b403-09fedab86a7c\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " Aug 04 18:38:23.393069 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.392949 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-workload-certs\") pod \"94683587-4d29-4f55-b403-09fedab86a7c\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " Aug 04 18:38:23.393069 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.392999 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-istio-envoy\") pod \"94683587-4d29-4f55-b403-09fedab86a7c\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " Aug 04 18:38:23.393069 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.393030 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8z7h\" (UniqueName: \"kubernetes.io/projected/94683587-4d29-4f55-b403-09fedab86a7c-kube-api-access-m8z7h\") pod \"94683587-4d29-4f55-b403-09fedab86a7c\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " Aug 04 18:38:23.393379 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.393127 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94683587-4d29-4f55-b403-09fedab86a7c-istio-podinfo\") pod \"94683587-4d29-4f55-b403-09fedab86a7c\" (UID: \"94683587-4d29-4f55-b403-09fedab86a7c\") " Aug 04 18:38:23.393379 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.393201 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94683587-4d29-4f55-b403-09fedab86a7c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "94683587-4d29-4f55-b403-09fedab86a7c" (UID: "94683587-4d29-4f55-b403-09fedab86a7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:38:23.393379 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.393262 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "94683587-4d29-4f55-b403-09fedab86a7c" (UID: "94683587-4d29-4f55-b403-09fedab86a7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:38:23.393379 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.393299 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "94683587-4d29-4f55-b403-09fedab86a7c" (UID: "94683587-4d29-4f55-b403-09fedab86a7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:38:23.393379 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.393319 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-istio-data" (OuterVolumeSpecName: "istio-data") pod "94683587-4d29-4f55-b403-09fedab86a7c" (UID: "94683587-4d29-4f55-b403-09fedab86a7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:38:23.393617 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.393494 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "94683587-4d29-4f55-b403-09fedab86a7c" (UID: "94683587-4d29-4f55-b403-09fedab86a7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:38:23.393617 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.393523 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94683587-4d29-4f55-b403-09fedab86a7c-istiod-ca-cert\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:38:23.393617 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.393544 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-istio-data\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:38:23.393617 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.393558 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-workload-socket\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:38:23.393617 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.393571 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-credential-socket\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:38:23.395265 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.395234 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94683587-4d29-4f55-b403-09fedab86a7c-istio-token" (OuterVolumeSpecName: "istio-token") pod "94683587-4d29-4f55-b403-09fedab86a7c" (UID: "94683587-4d29-4f55-b403-09fedab86a7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:38:23.395381 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.395283 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "94683587-4d29-4f55-b403-09fedab86a7c" (UID: "94683587-4d29-4f55-b403-09fedab86a7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:38:23.395851 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.395826 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94683587-4d29-4f55-b403-09fedab86a7c-kube-api-access-m8z7h" (OuterVolumeSpecName: "kube-api-access-m8z7h") pod "94683587-4d29-4f55-b403-09fedab86a7c" (UID: "94683587-4d29-4f55-b403-09fedab86a7c"). InnerVolumeSpecName "kube-api-access-m8z7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:38:23.395851 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.395831 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94683587-4d29-4f55-b403-09fedab86a7c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "94683587-4d29-4f55-b403-09fedab86a7c" (UID: "94683587-4d29-4f55-b403-09fedab86a7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:38:23.494407 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.494380 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-workload-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:38:23.494407 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.494403 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94683587-4d29-4f55-b403-09fedab86a7c-istio-envoy\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:38:23.494542 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.494416 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8z7h\" (UniqueName: \"kubernetes.io/projected/94683587-4d29-4f55-b403-09fedab86a7c-kube-api-access-m8z7h\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:38:23.494542 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.494429 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94683587-4d29-4f55-b403-09fedab86a7c-istio-podinfo\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:38:23.494542 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.494441 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94683587-4d29-4f55-b403-09fedab86a7c-istio-token\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:38:23.842232 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.842193 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:23.936874 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:23.936841 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:24.032092 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:24.032062 2572 generic.go:356] "Generic (PLEG): container finished" podID="94683587-4d29-4f55-b403-09fedab86a7c" containerID="6f16f8d2867017295a5d19151846dcc0ca920f202f072bf888e86c2f989ca45a" exitCode=0 Aug 04 18:38:24.032260 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:24.032113 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" Aug 04 18:38:24.032260 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:24.032131 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" event={"ID":"94683587-4d29-4f55-b403-09fedab86a7c","Type":"ContainerDied","Data":"6f16f8d2867017295a5d19151846dcc0ca920f202f072bf888e86c2f989ca45a"} Aug 04 18:38:24.032260 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:24.032175 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz" event={"ID":"94683587-4d29-4f55-b403-09fedab86a7c","Type":"ContainerDied","Data":"e005e991628d64376b4f15c05930cf6c8e2d5f37861de64ff82d9a47fcd0df80"} Aug 04 18:38:24.032260 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:24.032197 2572 scope.go:117] "RemoveContainer" containerID="6f16f8d2867017295a5d19151846dcc0ca920f202f072bf888e86c2f989ca45a" Aug 04 18:38:24.041979 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:24.041957 2572 scope.go:117] "RemoveContainer" containerID="23d72a728cb0c4e3fdae30c04dece297a953c779f5254dd7b3e56d0c8f90bc77" Aug 04 18:38:24.051487 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:24.051448 2572 scope.go:117] "RemoveContainer" containerID="6f16f8d2867017295a5d19151846dcc0ca920f202f072bf888e86c2f989ca45a" Aug 04 18:38:24.051869 ip-10-0-137-173 kubenswrapper[2572]: E0804 18:38:24.051839 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f16f8d2867017295a5d19151846dcc0ca920f202f072bf888e86c2f989ca45a\": container with ID starting with 6f16f8d2867017295a5d19151846dcc0ca920f202f072bf888e86c2f989ca45a not found: ID does not exist" containerID="6f16f8d2867017295a5d19151846dcc0ca920f202f072bf888e86c2f989ca45a" Aug 04 18:38:24.051980 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:24.051882 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f16f8d2867017295a5d19151846dcc0ca920f202f072bf888e86c2f989ca45a"} err="failed to get container status \"6f16f8d2867017295a5d19151846dcc0ca920f202f072bf888e86c2f989ca45a\": rpc error: code = NotFound desc = could not find container \"6f16f8d2867017295a5d19151846dcc0ca920f202f072bf888e86c2f989ca45a\": container with ID starting with 6f16f8d2867017295a5d19151846dcc0ca920f202f072bf888e86c2f989ca45a not found: ID does not exist" Aug 04 18:38:24.051980 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:24.051907 2572 scope.go:117] "RemoveContainer" containerID="23d72a728cb0c4e3fdae30c04dece297a953c779f5254dd7b3e56d0c8f90bc77" Aug 04 18:38:24.052371 ip-10-0-137-173 kubenswrapper[2572]: E0804 18:38:24.052340 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23d72a728cb0c4e3fdae30c04dece297a953c779f5254dd7b3e56d0c8f90bc77\": container with ID starting with 23d72a728cb0c4e3fdae30c04dece297a953c779f5254dd7b3e56d0c8f90bc77 not found: ID does not exist" containerID="23d72a728cb0c4e3fdae30c04dece297a953c779f5254dd7b3e56d0c8f90bc77" Aug 04 18:38:24.052466 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:24.052368 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d72a728cb0c4e3fdae30c04dece297a953c779f5254dd7b3e56d0c8f90bc77"} err="failed to get container status \"23d72a728cb0c4e3fdae30c04dece297a953c779f5254dd7b3e56d0c8f90bc77\": rpc error: code = NotFound desc = could not find container \"23d72a728cb0c4e3fdae30c04dece297a953c779f5254dd7b3e56d0c8f90bc77\": container with ID starting with 23d72a728cb0c4e3fdae30c04dece297a953c779f5254dd7b3e56d0c8f90bc77 not found: ID does not exist" Aug 04 18:38:24.054911 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:24.054890 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz"] Aug 04 18:38:24.060289 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:24.060270 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a77bad-openshift-default-54b4cd964b-v4ztz"] Aug 04 18:38:24.821747 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:24.821716 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94683587-4d29-4f55-b403-09fedab86a7c" path="/var/lib/kubelet/pods/94683587-4d29-4f55-b403-09fedab86a7c/volumes" Aug 04 18:38:36.017217 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:36.017181 2572 patch_prober.go:28] interesting pod/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Aug 04 18:38:36.017745 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:36.017262 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" podUID="3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Aug 04 18:38:36.092571 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:36.092538 2572 generic.go:356] "Generic (PLEG): container finished" podID="3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" containerID="356acf44522306aa3d55a8b6e95d07452f39aaf693dbaee13fb07f585d7726c1" exitCode=0 Aug 04 18:38:36.092791 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:36.092617 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" event={"ID":"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d","Type":"ContainerDied","Data":"356acf44522306aa3d55a8b6e95d07452f39aaf693dbaee13fb07f585d7726c1"} Aug 04 18:38:36.093076 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:36.093060 2572 scope.go:117] "RemoveContainer" containerID="356acf44522306aa3d55a8b6e95d07452f39aaf693dbaee13fb07f585d7726c1" Aug 04 18:38:36.094122 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:36.094096 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:38:36.094210 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:36.094161 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:38:36.094210 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:36.094191 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:38:37.098478 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:37.098445 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" event={"ID":"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d","Type":"ContainerStarted","Data":"b68a104185eb56cc3a11cdabd60bad2421f6e82ab8672630bd81f2d74c9606b5"} Aug 04 18:38:37.207549 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:37.207506 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:37.207796 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:37.207773 2572 patch_prober.go:28] interesting pod/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Aug 04 18:38:37.207849 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:37.207830 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" podUID="3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Aug 04 18:38:38.208288 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:38.208254 2572 patch_prober.go:28] interesting pod/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Aug 04 18:38:38.208775 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:38.208311 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" podUID="3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Aug 04 18:38:39.208625 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:39.208591 2572 patch_prober.go:28] interesting pod/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Aug 04 18:38:39.209021 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:39.208648 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" podUID="3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Aug 04 18:38:40.207999 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:40.207965 2572 patch_prober.go:28] interesting pod/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Aug 04 18:38:40.208178 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:40.208019 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" podUID="3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Aug 04 18:38:41.208239 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:41.208203 2572 patch_prober.go:28] interesting pod/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Aug 04 18:38:41.208610 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:41.208258 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" podUID="3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Aug 04 18:38:42.208581 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:42.208539 2572 patch_prober.go:28] interesting pod/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Aug 04 18:38:42.209088 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:42.208602 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" podUID="3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Aug 04 18:38:43.211470 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:43.211438 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:43.211896 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:43.211776 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:43.212557 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:43.212542 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:38:46.880028 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:46.879983 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:47.290081 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:47.290002 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:47.647218 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:47.647183 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:47.824651 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:47.824614 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:50.939857 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:50.939824 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:51.034258 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:51.034220 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:52.221190 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:52.221154 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:52.609121 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:52.609089 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:52.729358 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:52.729323 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:38:52.827025 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:38:52.826998 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:39:15.919078 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:15.919036 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:39:19.031169 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:19.031118 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:39:19.092008 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:19.091971 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:39:25.888696 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:25.888646 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp"] Aug 04 18:39:25.889055 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:25.889001 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94683587-4d29-4f55-b403-09fedab86a7c" containerName="istio-proxy" Aug 04 18:39:25.889055 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:25.889013 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="94683587-4d29-4f55-b403-09fedab86a7c" containerName="istio-proxy" Aug 04 18:39:25.889055 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:25.889026 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94683587-4d29-4f55-b403-09fedab86a7c" containerName="istio-proxy" Aug 04 18:39:25.889055 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:25.889032 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="94683587-4d29-4f55-b403-09fedab86a7c" containerName="istio-proxy" Aug 04 18:39:25.889190 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:25.889082 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="94683587-4d29-4f55-b403-09fedab86a7c" containerName="istio-proxy" Aug 04 18:39:25.892105 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:25.892091 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:39:25.895199 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:25.895181 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5c8685\"/\"openshift-service-ca.crt\"" Aug 04 18:39:25.896307 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:25.896285 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5c8685\"/\"kube-root-ca.crt\"" Aug 04 18:39:25.896428 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:25.896354 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5c8685\"/\"default-dockercfg-bfqmh\"" Aug 04 18:39:25.896428 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:25.896287 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5c8685\"/\"auth-test-model-5c8685-kserve-self-signed-certs\"" Aug 04 18:39:25.903688 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:25.903649 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp"] Aug 04 18:39:25.907555 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:25.907531 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:39:25.945046 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:25.945019 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-tls-certs\") pod \"auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp\" (UID: \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\") " pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:39:25.945211 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:25.945056 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z5z5\" (UniqueName: \"kubernetes.io/projected/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-kube-api-access-6z5z5\") pod \"auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp\" (UID: \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\") " pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:39:25.945211 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:25.945086 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-home\") pod \"auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp\" (UID: \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\") " pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:39:25.945211 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:25.945132 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-tmp-dir\") pod \"auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp\" (UID: \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\") " pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:39:25.945211 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:25.945159 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-dshm\") pod \"auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp\" (UID: \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\") " pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:39:25.945211 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:25.945177 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-model-cache\") pod \"auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp\" (UID: \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\") " pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:39:26.046463 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:26.046432 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z5z5\" (UniqueName: \"kubernetes.io/projected/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-kube-api-access-6z5z5\") pod \"auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp\" (UID: \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\") " pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:39:26.046627 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:26.046489 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-home\") pod \"auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp\" (UID: \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\") " pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:39:26.046627 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:26.046515 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-tmp-dir\") pod \"auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp\" (UID: \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\") " pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:39:26.046627 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:26.046546 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-dshm\") pod \"auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp\" (UID: \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\") " pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:39:26.046627 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:26.046576 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-model-cache\") pod \"auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp\" (UID: \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\") " pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:39:26.046878 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:26.046682 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-tls-certs\") pod \"auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp\" (UID: \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\") " pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:39:26.046961 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:26.046939 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-home\") pod \"auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp\" (UID: \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\") " pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:39:26.047034 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:26.046990 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-tmp-dir\") pod \"auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp\" (UID: \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\") " pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:39:26.047108 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:26.047087 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-model-cache\") pod \"auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp\" (UID: \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\") " pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:39:26.048866 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:26.048844 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-dshm\") pod \"auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp\" (UID: \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\") " pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:39:26.049204 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:26.049186 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-tls-certs\") pod \"auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp\" (UID: \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\") " pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:39:26.058750 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:26.058728 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z5z5\" (UniqueName: \"kubernetes.io/projected/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-kube-api-access-6z5z5\") pod \"auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp\" (UID: \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\") " pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:39:26.201421 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:26.201339 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:39:26.332014 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:26.331991 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp"] Aug 04 18:39:26.333442 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:39:26.333419 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9d6a609_4ef4_4e6d_b6fd_b5f9357edc2d.slice/crio-486709d6b7565b94b13bdd24bc5dbc26c816dbf6be1ee836732f601a451b1dd7 WatchSource:0}: Error finding container 486709d6b7565b94b13bdd24bc5dbc26c816dbf6be1ee836732f601a451b1dd7: Status 404 returned error can't find the container with id 486709d6b7565b94b13bdd24bc5dbc26c816dbf6be1ee836732f601a451b1dd7 Aug 04 18:39:27.331044 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:27.331013 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" event={"ID":"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d","Type":"ContainerStarted","Data":"4ddb02cb5213deb6654806fe44eed02d09122848e10cb08272d40b9dbf78b3d4"} Aug 04 18:39:27.331431 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:27.331051 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" event={"ID":"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d","Type":"ContainerStarted","Data":"486709d6b7565b94b13bdd24bc5dbc26c816dbf6be1ee836732f601a451b1dd7"} Aug 04 18:39:27.353789 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:27.353743 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" podStartSLOduration=2.353726989 podStartE2EDuration="2.353726989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:39:27.352417045 +0000 UTC m=+2823.043108095" watchObservedRunningTime="2026-08-04 18:39:27.353726989 +0000 UTC m=+2823.044418040" Aug 04 18:39:36.202031 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:36.201989 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:39:36.202437 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:36.202043 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:39:36.214499 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:36.214476 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:39:36.389313 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:36.389285 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:39:40.690001 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.689967 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj"] Aug 04 18:39:40.690390 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.690370 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="94683587-4d29-4f55-b403-09fedab86a7c" containerName="istio-proxy" Aug 04 18:39:40.693346 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.693331 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:39:40.697407 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.697379 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc18f1\"/\"openshift-service-ca.crt\"" Aug 04 18:39:40.697407 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.697396 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc18f1\"/\"default-dockercfg-mgdn7\"" Aug 04 18:39:40.697407 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.697384 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc18f1\"/\"auth-test-model-bc18f1-kserve-self-signed-certs\"" Aug 04 18:39:40.697705 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.697689 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc18f1\"/\"kube-root-ca.crt\"" Aug 04 18:39:40.703658 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.703636 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj"] Aug 04 18:39:40.714834 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.714812 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:39:40.775901 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.775870 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcms4\" (UniqueName: \"kubernetes.io/projected/af0295c0-808d-4dca-8628-9509bf858059-kube-api-access-rcms4\") pod \"auth-test-model-bc18f1-kserve-586577bcd7-jx9mj\" (UID: \"af0295c0-808d-4dca-8628-9509bf858059\") " pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:39:40.776051 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.775912 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af0295c0-808d-4dca-8628-9509bf858059-home\") pod \"auth-test-model-bc18f1-kserve-586577bcd7-jx9mj\" (UID: \"af0295c0-808d-4dca-8628-9509bf858059\") " pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:39:40.776051 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.775995 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af0295c0-808d-4dca-8628-9509bf858059-dshm\") pod \"auth-test-model-bc18f1-kserve-586577bcd7-jx9mj\" (UID: \"af0295c0-808d-4dca-8628-9509bf858059\") " pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:39:40.776051 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.776032 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af0295c0-808d-4dca-8628-9509bf858059-tls-certs\") pod \"auth-test-model-bc18f1-kserve-586577bcd7-jx9mj\" (UID: \"af0295c0-808d-4dca-8628-9509bf858059\") " pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:39:40.776200 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.776052 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af0295c0-808d-4dca-8628-9509bf858059-model-cache\") pod \"auth-test-model-bc18f1-kserve-586577bcd7-jx9mj\" (UID: \"af0295c0-808d-4dca-8628-9509bf858059\") " pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:39:40.776200 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.776088 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af0295c0-808d-4dca-8628-9509bf858059-tmp-dir\") pod \"auth-test-model-bc18f1-kserve-586577bcd7-jx9mj\" (UID: \"af0295c0-808d-4dca-8628-9509bf858059\") " pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:39:40.877046 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.877013 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcms4\" (UniqueName: \"kubernetes.io/projected/af0295c0-808d-4dca-8628-9509bf858059-kube-api-access-rcms4\") pod \"auth-test-model-bc18f1-kserve-586577bcd7-jx9mj\" (UID: \"af0295c0-808d-4dca-8628-9509bf858059\") " pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:39:40.877219 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.877053 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af0295c0-808d-4dca-8628-9509bf858059-home\") pod \"auth-test-model-bc18f1-kserve-586577bcd7-jx9mj\" (UID: \"af0295c0-808d-4dca-8628-9509bf858059\") " pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:39:40.877219 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.877087 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af0295c0-808d-4dca-8628-9509bf858059-dshm\") pod \"auth-test-model-bc18f1-kserve-586577bcd7-jx9mj\" (UID: \"af0295c0-808d-4dca-8628-9509bf858059\") " pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:39:40.877219 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.877109 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af0295c0-808d-4dca-8628-9509bf858059-tls-certs\") pod \"auth-test-model-bc18f1-kserve-586577bcd7-jx9mj\" (UID: \"af0295c0-808d-4dca-8628-9509bf858059\") " pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:39:40.877219 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.877129 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af0295c0-808d-4dca-8628-9509bf858059-model-cache\") pod \"auth-test-model-bc18f1-kserve-586577bcd7-jx9mj\" (UID: \"af0295c0-808d-4dca-8628-9509bf858059\") " pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:39:40.877219 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.877148 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af0295c0-808d-4dca-8628-9509bf858059-tmp-dir\") pod \"auth-test-model-bc18f1-kserve-586577bcd7-jx9mj\" (UID: \"af0295c0-808d-4dca-8628-9509bf858059\") " pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:39:40.877503 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.877484 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af0295c0-808d-4dca-8628-9509bf858059-home\") pod \"auth-test-model-bc18f1-kserve-586577bcd7-jx9mj\" (UID: \"af0295c0-808d-4dca-8628-9509bf858059\") " pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:39:40.877555 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.877533 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af0295c0-808d-4dca-8628-9509bf858059-tmp-dir\") pod \"auth-test-model-bc18f1-kserve-586577bcd7-jx9mj\" (UID: \"af0295c0-808d-4dca-8628-9509bf858059\") " pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:39:40.877617 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.877551 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af0295c0-808d-4dca-8628-9509bf858059-model-cache\") pod \"auth-test-model-bc18f1-kserve-586577bcd7-jx9mj\" (UID: \"af0295c0-808d-4dca-8628-9509bf858059\") " pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:39:40.879696 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.879653 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af0295c0-808d-4dca-8628-9509bf858059-dshm\") pod \"auth-test-model-bc18f1-kserve-586577bcd7-jx9mj\" (UID: \"af0295c0-808d-4dca-8628-9509bf858059\") " pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:39:40.879820 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.879806 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af0295c0-808d-4dca-8628-9509bf858059-tls-certs\") pod \"auth-test-model-bc18f1-kserve-586577bcd7-jx9mj\" (UID: \"af0295c0-808d-4dca-8628-9509bf858059\") " pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:39:40.887103 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:40.887081 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcms4\" (UniqueName: \"kubernetes.io/projected/af0295c0-808d-4dca-8628-9509bf858059-kube-api-access-rcms4\") pod \"auth-test-model-bc18f1-kserve-586577bcd7-jx9mj\" (UID: \"af0295c0-808d-4dca-8628-9509bf858059\") " pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:39:41.003135 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:41.003030 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:39:41.132522 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:41.132496 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj"] Aug 04 18:39:41.134924 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:39:41.134896 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf0295c0_808d_4dca_8628_9509bf858059.slice/crio-0912d08caa7884e41a7f40b55e3868c7a9d0e3e976bb73b19b06c1cd58ac7528 WatchSource:0}: Error finding container 0912d08caa7884e41a7f40b55e3868c7a9d0e3e976bb73b19b06c1cd58ac7528: Status 404 returned error can't find the container with id 0912d08caa7884e41a7f40b55e3868c7a9d0e3e976bb73b19b06c1cd58ac7528 Aug 04 18:39:41.402450 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:41.402414 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" event={"ID":"af0295c0-808d-4dca-8628-9509bf858059","Type":"ContainerStarted","Data":"02ff77f696b7fe12a0da01fbdad7340f679f2617ceb38a90d1d674a972dfecfc"} Aug 04 18:39:41.402618 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:41.402457 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" event={"ID":"af0295c0-808d-4dca-8628-9509bf858059","Type":"ContainerStarted","Data":"0912d08caa7884e41a7f40b55e3868c7a9d0e3e976bb73b19b06c1cd58ac7528"} Aug 04 18:39:41.424215 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:41.424173 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" podStartSLOduration=1.424158512 podStartE2EDuration="1.424158512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:39:41.422392468 +0000 UTC m=+2837.113083518" watchObservedRunningTime="2026-08-04 18:39:41.424158512 +0000 UTC m=+2837.114849562" Aug 04 18:39:51.004139 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:51.004091 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:39:51.004139 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:51.004149 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:39:51.017103 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:51.017077 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:39:51.459056 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:39:51.459021 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:40:11.562074 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:40:11.562031 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:40:52.978939 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:40:52.978903 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:42:21.336782 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:21.336748 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:42:25.106634 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:25.106599 2572 scope.go:117] "RemoveContainer" containerID="f1ac4b9ec830bc568f7a82689e374627516d23964a1e0806ea0c4f723aad1e2f" Aug 04 18:42:39.145620 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:39.145589 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp"] Aug 04 18:42:39.169479 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:39.169448 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:42:40.009005 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:40.008975 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj"] Aug 04 18:42:40.029944 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:40.029896 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:42:47.004822 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.004792 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf"] Aug 04 18:42:47.008238 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.008221 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:42:47.011444 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.011428 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5c8685\"/\"sub-test-model-5c8685-kserve-self-signed-certs\"" Aug 04 18:42:47.023011 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.022991 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf"] Aug 04 18:42:47.087420 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.087385 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:42:47.090582 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.090563 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aadef35-91ef-43d9-b10e-18be4abc0110-tmp-dir\") pod \"sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf\" (UID: \"7aadef35-91ef-43d9-b10e-18be4abc0110\") " pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:42:47.090656 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.090597 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7aadef35-91ef-43d9-b10e-18be4abc0110-tls-certs\") pod \"sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf\" (UID: \"7aadef35-91ef-43d9-b10e-18be4abc0110\") " pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:42:47.090656 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.090621 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7aadef35-91ef-43d9-b10e-18be4abc0110-model-cache\") pod \"sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf\" (UID: \"7aadef35-91ef-43d9-b10e-18be4abc0110\") " pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:42:47.090760 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.090700 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7aadef35-91ef-43d9-b10e-18be4abc0110-home\") pod \"sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf\" (UID: \"7aadef35-91ef-43d9-b10e-18be4abc0110\") " pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:42:47.090760 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.090751 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj97s\" (UniqueName: \"kubernetes.io/projected/7aadef35-91ef-43d9-b10e-18be4abc0110-kube-api-access-zj97s\") pod \"sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf\" (UID: \"7aadef35-91ef-43d9-b10e-18be4abc0110\") " pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:42:47.090845 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.090827 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7aadef35-91ef-43d9-b10e-18be4abc0110-dshm\") pod \"sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf\" (UID: \"7aadef35-91ef-43d9-b10e-18be4abc0110\") " pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:42:47.191588 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.191551 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7aadef35-91ef-43d9-b10e-18be4abc0110-dshm\") pod \"sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf\" (UID: \"7aadef35-91ef-43d9-b10e-18be4abc0110\") " pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:42:47.191800 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.191614 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aadef35-91ef-43d9-b10e-18be4abc0110-tmp-dir\") pod \"sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf\" (UID: \"7aadef35-91ef-43d9-b10e-18be4abc0110\") " pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:42:47.191800 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.191638 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7aadef35-91ef-43d9-b10e-18be4abc0110-tls-certs\") pod \"sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf\" (UID: \"7aadef35-91ef-43d9-b10e-18be4abc0110\") " pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:42:47.191800 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.191684 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7aadef35-91ef-43d9-b10e-18be4abc0110-model-cache\") pod \"sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf\" (UID: \"7aadef35-91ef-43d9-b10e-18be4abc0110\") " pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:42:47.191800 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.191710 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7aadef35-91ef-43d9-b10e-18be4abc0110-home\") pod \"sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf\" (UID: \"7aadef35-91ef-43d9-b10e-18be4abc0110\") " pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:42:47.191800 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.191728 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj97s\" (UniqueName: \"kubernetes.io/projected/7aadef35-91ef-43d9-b10e-18be4abc0110-kube-api-access-zj97s\") pod \"sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf\" (UID: \"7aadef35-91ef-43d9-b10e-18be4abc0110\") " pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:42:47.192176 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.192147 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aadef35-91ef-43d9-b10e-18be4abc0110-tmp-dir\") pod \"sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf\" (UID: \"7aadef35-91ef-43d9-b10e-18be4abc0110\") " pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:42:47.192260 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.192176 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7aadef35-91ef-43d9-b10e-18be4abc0110-model-cache\") pod \"sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf\" (UID: \"7aadef35-91ef-43d9-b10e-18be4abc0110\") " pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:42:47.192260 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.192205 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7aadef35-91ef-43d9-b10e-18be4abc0110-home\") pod \"sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf\" (UID: \"7aadef35-91ef-43d9-b10e-18be4abc0110\") " pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:42:47.194078 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.194060 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7aadef35-91ef-43d9-b10e-18be4abc0110-dshm\") pod \"sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf\" (UID: \"7aadef35-91ef-43d9-b10e-18be4abc0110\") " pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:42:47.194414 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.194396 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7aadef35-91ef-43d9-b10e-18be4abc0110-tls-certs\") pod \"sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf\" (UID: \"7aadef35-91ef-43d9-b10e-18be4abc0110\") " pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:42:47.203257 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.203232 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj97s\" (UniqueName: \"kubernetes.io/projected/7aadef35-91ef-43d9-b10e-18be4abc0110-kube-api-access-zj97s\") pod \"sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf\" (UID: \"7aadef35-91ef-43d9-b10e-18be4abc0110\") " pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:42:47.317659 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.317635 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:42:47.453736 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:47.453706 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf"] Aug 04 18:42:47.456223 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:42:47.456183 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7aadef35_91ef_43d9_b10e_18be4abc0110.slice/crio-36fb454d668af13bca80f2eed4ed888abeea439c5fe76f59d4dabee832072df9 WatchSource:0}: Error finding container 36fb454d668af13bca80f2eed4ed888abeea439c5fe76f59d4dabee832072df9: Status 404 returned error can't find the container with id 36fb454d668af13bca80f2eed4ed888abeea439c5fe76f59d4dabee832072df9 Aug 04 18:42:48.301936 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:48.301905 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" event={"ID":"7aadef35-91ef-43d9-b10e-18be4abc0110","Type":"ContainerStarted","Data":"82163d57103f779676b40a8890152c89d6b56bab55d9e6d716aa469dd17f9c36"} Aug 04 18:42:48.302397 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:48.301952 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" event={"ID":"7aadef35-91ef-43d9-b10e-18be4abc0110","Type":"ContainerStarted","Data":"36fb454d668af13bca80f2eed4ed888abeea439c5fe76f59d4dabee832072df9"} Aug 04 18:42:48.320211 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:48.320165 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" podStartSLOduration=2.320149052 podStartE2EDuration="2.320149052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:42:48.318965256 +0000 UTC m=+3024.009656308" watchObservedRunningTime="2026-08-04 18:42:48.320149052 +0000 UTC m=+3024.010840140" Aug 04 18:42:54.164743 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:54.164704 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" podUID="c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d" containerName="main" containerID="cri-o://4ddb02cb5213deb6654806fe44eed02d09122848e10cb08272d40b9dbf78b3d4" gracePeriod=45 Aug 04 18:42:54.331951 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:54.331922 2572 generic.go:356] "Generic (PLEG): container finished" podID="c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d" containerID="4ddb02cb5213deb6654806fe44eed02d09122848e10cb08272d40b9dbf78b3d4" exitCode=0 Aug 04 18:42:54.332102 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:54.331980 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" event={"ID":"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d","Type":"ContainerDied","Data":"4ddb02cb5213deb6654806fe44eed02d09122848e10cb08272d40b9dbf78b3d4"} Aug 04 18:42:54.415952 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:54.415896 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:42:54.560304 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:54.560275 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z5z5\" (UniqueName: \"kubernetes.io/projected/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-kube-api-access-6z5z5\") pod \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\" (UID: \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\") " Aug 04 18:42:54.560488 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:54.560343 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-tmp-dir\") pod \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\" (UID: \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\") " Aug 04 18:42:54.560488 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:54.560362 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-home\") pod \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\" (UID: \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\") " Aug 04 18:42:54.560613 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:54.560519 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-dshm\") pod \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\" (UID: \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\") " Aug 04 18:42:54.560613 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:54.560563 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-model-cache\") pod \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\" (UID: \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\") " Aug 04 18:42:54.560613 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:54.560595 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-home" (OuterVolumeSpecName: "home") pod "c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d" (UID: "c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:42:54.560782 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:54.560594 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d" (UID: "c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:42:54.560782 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:54.560612 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-tls-certs\") pod \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\" (UID: \"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d\") " Aug 04 18:42:54.560877 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:54.560798 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d" (UID: "c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:42:54.561270 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:54.561245 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-tmp-dir\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:42:54.561439 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:54.561418 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-home\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:42:54.561521 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:54.561444 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-model-cache\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:42:54.562969 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:54.562947 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d" (UID: "c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:42:54.563059 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:54.562965 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-dshm" (OuterVolumeSpecName: "dshm") pod "c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d" (UID: "c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:42:54.563059 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:54.562996 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-kube-api-access-6z5z5" (OuterVolumeSpecName: "kube-api-access-6z5z5") pod "c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d" (UID: "c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d"). InnerVolumeSpecName "kube-api-access-6z5z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:42:54.662516 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:54.662486 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z5z5\" (UniqueName: \"kubernetes.io/projected/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-kube-api-access-6z5z5\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:42:54.662516 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:54.662513 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-dshm\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:42:54.662736 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:54.662522 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d-tls-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:42:55.028117 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.028061 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" podUID="af0295c0-808d-4dca-8628-9509bf858059" containerName="main" containerID="cri-o://02ff77f696b7fe12a0da01fbdad7340f679f2617ceb38a90d1d674a972dfecfc" gracePeriod=45 Aug 04 18:42:55.268018 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.267988 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:42:55.337980 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.337938 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" Aug 04 18:42:55.338173 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.337939 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp" event={"ID":"c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d","Type":"ContainerDied","Data":"486709d6b7565b94b13bdd24bc5dbc26c816dbf6be1ee836732f601a451b1dd7"} Aug 04 18:42:55.338173 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.338057 2572 scope.go:117] "RemoveContainer" containerID="4ddb02cb5213deb6654806fe44eed02d09122848e10cb08272d40b9dbf78b3d4" Aug 04 18:42:55.339470 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.339444 2572 generic.go:356] "Generic (PLEG): container finished" podID="af0295c0-808d-4dca-8628-9509bf858059" containerID="02ff77f696b7fe12a0da01fbdad7340f679f2617ceb38a90d1d674a972dfecfc" exitCode=0 Aug 04 18:42:55.339595 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.339499 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" Aug 04 18:42:55.339595 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.339539 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" event={"ID":"af0295c0-808d-4dca-8628-9509bf858059","Type":"ContainerDied","Data":"02ff77f696b7fe12a0da01fbdad7340f679f2617ceb38a90d1d674a972dfecfc"} Aug 04 18:42:55.339595 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.339575 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj" event={"ID":"af0295c0-808d-4dca-8628-9509bf858059","Type":"ContainerDied","Data":"0912d08caa7884e41a7f40b55e3868c7a9d0e3e976bb73b19b06c1cd58ac7528"} Aug 04 18:42:55.348100 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.348073 2572 scope.go:117] "RemoveContainer" containerID="02ff77f696b7fe12a0da01fbdad7340f679f2617ceb38a90d1d674a972dfecfc" Aug 04 18:42:55.355804 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.355787 2572 scope.go:117] "RemoveContainer" containerID="02ff77f696b7fe12a0da01fbdad7340f679f2617ceb38a90d1d674a972dfecfc" Aug 04 18:42:55.356099 ip-10-0-137-173 kubenswrapper[2572]: E0804 18:42:55.356073 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02ff77f696b7fe12a0da01fbdad7340f679f2617ceb38a90d1d674a972dfecfc\": container with ID starting with 02ff77f696b7fe12a0da01fbdad7340f679f2617ceb38a90d1d674a972dfecfc not found: ID does not exist" containerID="02ff77f696b7fe12a0da01fbdad7340f679f2617ceb38a90d1d674a972dfecfc" Aug 04 18:42:55.356154 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.356103 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02ff77f696b7fe12a0da01fbdad7340f679f2617ceb38a90d1d674a972dfecfc"} err="failed to get container status \"02ff77f696b7fe12a0da01fbdad7340f679f2617ceb38a90d1d674a972dfecfc\": rpc error: code = NotFound desc = could not find container \"02ff77f696b7fe12a0da01fbdad7340f679f2617ceb38a90d1d674a972dfecfc\": container with ID starting with 02ff77f696b7fe12a0da01fbdad7340f679f2617ceb38a90d1d674a972dfecfc not found: ID does not exist" Aug 04 18:42:55.357486 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.357468 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp"] Aug 04 18:42:55.361433 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.361414 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-5c8685/auth-test-model-5c8685-kserve-6464d6f5dc-x9gxp"] Aug 04 18:42:55.368588 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.368563 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcms4\" (UniqueName: \"kubernetes.io/projected/af0295c0-808d-4dca-8628-9509bf858059-kube-api-access-rcms4\") pod \"af0295c0-808d-4dca-8628-9509bf858059\" (UID: \"af0295c0-808d-4dca-8628-9509bf858059\") " Aug 04 18:42:55.368723 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.368600 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af0295c0-808d-4dca-8628-9509bf858059-model-cache\") pod \"af0295c0-808d-4dca-8628-9509bf858059\" (UID: \"af0295c0-808d-4dca-8628-9509bf858059\") " Aug 04 18:42:55.368723 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.368621 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af0295c0-808d-4dca-8628-9509bf858059-tls-certs\") pod \"af0295c0-808d-4dca-8628-9509bf858059\" (UID: \"af0295c0-808d-4dca-8628-9509bf858059\") " Aug 04 18:42:55.368842 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.368721 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af0295c0-808d-4dca-8628-9509bf858059-home\") pod \"af0295c0-808d-4dca-8628-9509bf858059\" (UID: \"af0295c0-808d-4dca-8628-9509bf858059\") " Aug 04 18:42:55.368842 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.368747 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af0295c0-808d-4dca-8628-9509bf858059-tmp-dir\") pod \"af0295c0-808d-4dca-8628-9509bf858059\" (UID: \"af0295c0-808d-4dca-8628-9509bf858059\") " Aug 04 18:42:55.368842 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.368772 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af0295c0-808d-4dca-8628-9509bf858059-dshm\") pod \"af0295c0-808d-4dca-8628-9509bf858059\" (UID: \"af0295c0-808d-4dca-8628-9509bf858059\") " Aug 04 18:42:55.368990 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.368883 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0295c0-808d-4dca-8628-9509bf858059-model-cache" (OuterVolumeSpecName: "model-cache") pod "af0295c0-808d-4dca-8628-9509bf858059" (UID: "af0295c0-808d-4dca-8628-9509bf858059"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:42:55.369057 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.368986 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0295c0-808d-4dca-8628-9509bf858059-home" (OuterVolumeSpecName: "home") pod "af0295c0-808d-4dca-8628-9509bf858059" (UID: "af0295c0-808d-4dca-8628-9509bf858059"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:42:55.369057 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.369012 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0295c0-808d-4dca-8628-9509bf858059-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "af0295c0-808d-4dca-8628-9509bf858059" (UID: "af0295c0-808d-4dca-8628-9509bf858059"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:42:55.369163 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.369122 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af0295c0-808d-4dca-8628-9509bf858059-model-cache\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:42:55.369163 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.369140 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af0295c0-808d-4dca-8628-9509bf858059-home\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:42:55.369163 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.369149 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af0295c0-808d-4dca-8628-9509bf858059-tmp-dir\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:42:55.370845 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.370823 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af0295c0-808d-4dca-8628-9509bf858059-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "af0295c0-808d-4dca-8628-9509bf858059" (UID: "af0295c0-808d-4dca-8628-9509bf858059"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:42:55.370964 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.370946 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af0295c0-808d-4dca-8628-9509bf858059-kube-api-access-rcms4" (OuterVolumeSpecName: "kube-api-access-rcms4") pod "af0295c0-808d-4dca-8628-9509bf858059" (UID: "af0295c0-808d-4dca-8628-9509bf858059"). InnerVolumeSpecName "kube-api-access-rcms4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:42:55.371010 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.370946 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0295c0-808d-4dca-8628-9509bf858059-dshm" (OuterVolumeSpecName: "dshm") pod "af0295c0-808d-4dca-8628-9509bf858059" (UID: "af0295c0-808d-4dca-8628-9509bf858059"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:42:55.470199 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.470132 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcms4\" (UniqueName: \"kubernetes.io/projected/af0295c0-808d-4dca-8628-9509bf858059-kube-api-access-rcms4\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:42:55.470199 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.470164 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af0295c0-808d-4dca-8628-9509bf858059-tls-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:42:55.470199 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.470173 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af0295c0-808d-4dca-8628-9509bf858059-dshm\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:42:55.662901 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.662873 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj"] Aug 04 18:42:55.668043 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:55.668015 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bc18f1/auth-test-model-bc18f1-kserve-586577bcd7-jx9mj"] Aug 04 18:42:56.821943 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:56.821912 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af0295c0-808d-4dca-8628-9509bf858059" path="/var/lib/kubelet/pods/af0295c0-808d-4dca-8628-9509bf858059/volumes" Aug 04 18:42:56.822306 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:56.822296 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d" path="/var/lib/kubelet/pods/c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d/volumes" Aug 04 18:42:57.318294 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:57.318271 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:42:57.318586 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:57.318569 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:42:57.330852 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:57.330832 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:42:57.362765 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:42:57.362744 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:43:06.291239 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.291202 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz"] Aug 04 18:43:06.291802 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.291736 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af0295c0-808d-4dca-8628-9509bf858059" containerName="main" Aug 04 18:43:06.291802 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.291754 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="af0295c0-808d-4dca-8628-9509bf858059" containerName="main" Aug 04 18:43:06.291802 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.291778 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d" containerName="main" Aug 04 18:43:06.291802 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.291785 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d" containerName="main" Aug 04 18:43:06.292038 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.291892 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9d6a609-4ef4-4e6d-b6fd-b5f9357edc2d" containerName="main" Aug 04 18:43:06.292038 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.291904 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="af0295c0-808d-4dca-8628-9509bf858059" containerName="main" Aug 04 18:43:06.294905 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.294883 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:43:06.297746 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.297727 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc18f1\"/\"kube-root-ca.crt\"" Aug 04 18:43:06.298690 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.298646 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc18f1\"/\"openshift-service-ca.crt\"" Aug 04 18:43:06.298854 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.298827 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc18f1\"/\"default-dockercfg-mgdn7\"" Aug 04 18:43:06.298973 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.298879 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc18f1\"/\"sub-test-model-bc18f1-kserve-self-signed-certs\"" Aug 04 18:43:06.304764 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.304745 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz"] Aug 04 18:43:06.373708 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.373651 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:43:06.471472 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.471440 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b53717-fe8f-469d-99e9-577e55477c8e-model-cache\") pod \"sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz\" (UID: \"21b53717-fe8f-469d-99e9-577e55477c8e\") " pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:43:06.471638 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.471481 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b53717-fe8f-469d-99e9-577e55477c8e-dshm\") pod \"sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz\" (UID: \"21b53717-fe8f-469d-99e9-577e55477c8e\") " pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:43:06.471638 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.471505 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b53717-fe8f-469d-99e9-577e55477c8e-home\") pod \"sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz\" (UID: \"21b53717-fe8f-469d-99e9-577e55477c8e\") " pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:43:06.471638 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.471545 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kckv\" (UniqueName: \"kubernetes.io/projected/21b53717-fe8f-469d-99e9-577e55477c8e-kube-api-access-7kckv\") pod \"sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz\" (UID: \"21b53717-fe8f-469d-99e9-577e55477c8e\") " pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:43:06.471638 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.471595 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b53717-fe8f-469d-99e9-577e55477c8e-tls-certs\") pod \"sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz\" (UID: \"21b53717-fe8f-469d-99e9-577e55477c8e\") " pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:43:06.471871 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.471650 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b53717-fe8f-469d-99e9-577e55477c8e-tmp-dir\") pod \"sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz\" (UID: \"21b53717-fe8f-469d-99e9-577e55477c8e\") " pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:43:06.572509 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.572473 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b53717-fe8f-469d-99e9-577e55477c8e-model-cache\") pod \"sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz\" (UID: \"21b53717-fe8f-469d-99e9-577e55477c8e\") " pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:43:06.572703 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.572518 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b53717-fe8f-469d-99e9-577e55477c8e-dshm\") pod \"sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz\" (UID: \"21b53717-fe8f-469d-99e9-577e55477c8e\") " pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:43:06.572703 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.572534 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b53717-fe8f-469d-99e9-577e55477c8e-home\") pod \"sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz\" (UID: \"21b53717-fe8f-469d-99e9-577e55477c8e\") " pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:43:06.572703 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.572565 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kckv\" (UniqueName: \"kubernetes.io/projected/21b53717-fe8f-469d-99e9-577e55477c8e-kube-api-access-7kckv\") pod \"sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz\" (UID: \"21b53717-fe8f-469d-99e9-577e55477c8e\") " pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:43:06.572703 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.572636 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b53717-fe8f-469d-99e9-577e55477c8e-tls-certs\") pod \"sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz\" (UID: \"21b53717-fe8f-469d-99e9-577e55477c8e\") " pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:43:06.572703 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.572685 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b53717-fe8f-469d-99e9-577e55477c8e-tmp-dir\") pod \"sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz\" (UID: \"21b53717-fe8f-469d-99e9-577e55477c8e\") " pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:43:06.572975 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.572950 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b53717-fe8f-469d-99e9-577e55477c8e-model-cache\") pod \"sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz\" (UID: \"21b53717-fe8f-469d-99e9-577e55477c8e\") " pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:43:06.573020 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.572988 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b53717-fe8f-469d-99e9-577e55477c8e-home\") pod \"sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz\" (UID: \"21b53717-fe8f-469d-99e9-577e55477c8e\") " pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:43:06.574263 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.573505 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b53717-fe8f-469d-99e9-577e55477c8e-tmp-dir\") pod \"sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz\" (UID: \"21b53717-fe8f-469d-99e9-577e55477c8e\") " pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:43:06.580144 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.580123 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b53717-fe8f-469d-99e9-577e55477c8e-dshm\") pod \"sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz\" (UID: \"21b53717-fe8f-469d-99e9-577e55477c8e\") " pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:43:06.580261 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.580209 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b53717-fe8f-469d-99e9-577e55477c8e-tls-certs\") pod \"sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz\" (UID: \"21b53717-fe8f-469d-99e9-577e55477c8e\") " pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:43:06.582528 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.582501 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kckv\" (UniqueName: \"kubernetes.io/projected/21b53717-fe8f-469d-99e9-577e55477c8e-kube-api-access-7kckv\") pod \"sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz\" (UID: \"21b53717-fe8f-469d-99e9-577e55477c8e\") " pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:43:06.604022 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.603994 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:43:06.736437 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:06.736405 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz"] Aug 04 18:43:06.739815 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:43:06.739789 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21b53717_fe8f_469d_99e9_577e55477c8e.slice/crio-0f7666bd5d3821379ae65384e63df6431c21eb4a97032f2011b5164419ae39b3 WatchSource:0}: Error finding container 0f7666bd5d3821379ae65384e63df6431c21eb4a97032f2011b5164419ae39b3: Status 404 returned error can't find the container with id 0f7666bd5d3821379ae65384e63df6431c21eb4a97032f2011b5164419ae39b3 Aug 04 18:43:07.399167 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:07.399133 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" event={"ID":"21b53717-fe8f-469d-99e9-577e55477c8e","Type":"ContainerStarted","Data":"6c686a702720a671d9d08ea0357c8f593c4f3a8fcebd2185fe1342c826078ba5"} Aug 04 18:43:07.399167 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:07.399172 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" event={"ID":"21b53717-fe8f-469d-99e9-577e55477c8e","Type":"ContainerStarted","Data":"0f7666bd5d3821379ae65384e63df6431c21eb4a97032f2011b5164419ae39b3"} Aug 04 18:43:07.417161 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:07.417090 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" podStartSLOduration=1.417073126 podStartE2EDuration="1.417073126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:43:07.416817553 +0000 UTC m=+3043.107508603" watchObservedRunningTime="2026-08-04 18:43:07.417073126 +0000 UTC m=+3043.107764177" Aug 04 18:43:16.604801 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:16.604712 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:43:16.604801 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:16.604772 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:43:16.617046 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:16.617010 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:43:17.457731 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:43:17.457703 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:44:17.833972 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:17.833943 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf"] Aug 04 18:44:17.851018 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:17.850987 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:44:18.683265 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:18.683232 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz"] Aug 04 18:44:18.699306 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:18.699278 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:44:25.143318 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:25.143285 2572 scope.go:117] "RemoveContainer" containerID="9e8afcc72d4669946046564fef3c83402a0d16292ad100ac7499b8f819327e1d" Aug 04 18:44:25.151370 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:25.151344 2572 scope.go:117] "RemoveContainer" containerID="a39e4a4d763963a26b2d983105c388c65dac956ac98ed803fa0dacaeaa9d68b7" Aug 04 18:44:32.852775 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:32.852736 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" podUID="7aadef35-91ef-43d9-b10e-18be4abc0110" containerName="main" containerID="cri-o://82163d57103f779676b40a8890152c89d6b56bab55d9e6d716aa469dd17f9c36" gracePeriod=45 Aug 04 18:44:33.095103 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.095078 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:44:33.234397 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.234303 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7aadef35-91ef-43d9-b10e-18be4abc0110-model-cache\") pod \"7aadef35-91ef-43d9-b10e-18be4abc0110\" (UID: \"7aadef35-91ef-43d9-b10e-18be4abc0110\") " Aug 04 18:44:33.234397 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.234370 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7aadef35-91ef-43d9-b10e-18be4abc0110-home\") pod \"7aadef35-91ef-43d9-b10e-18be4abc0110\" (UID: \"7aadef35-91ef-43d9-b10e-18be4abc0110\") " Aug 04 18:44:33.234397 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.234399 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aadef35-91ef-43d9-b10e-18be4abc0110-tmp-dir\") pod \"7aadef35-91ef-43d9-b10e-18be4abc0110\" (UID: \"7aadef35-91ef-43d9-b10e-18be4abc0110\") " Aug 04 18:44:33.234710 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.234434 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7aadef35-91ef-43d9-b10e-18be4abc0110-dshm\") pod \"7aadef35-91ef-43d9-b10e-18be4abc0110\" (UID: \"7aadef35-91ef-43d9-b10e-18be4abc0110\") " Aug 04 18:44:33.234710 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.234464 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7aadef35-91ef-43d9-b10e-18be4abc0110-tls-certs\") pod \"7aadef35-91ef-43d9-b10e-18be4abc0110\" (UID: \"7aadef35-91ef-43d9-b10e-18be4abc0110\") " Aug 04 18:44:33.234710 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.234525 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zj97s\" (UniqueName: \"kubernetes.io/projected/7aadef35-91ef-43d9-b10e-18be4abc0110-kube-api-access-zj97s\") pod \"7aadef35-91ef-43d9-b10e-18be4abc0110\" (UID: \"7aadef35-91ef-43d9-b10e-18be4abc0110\") " Aug 04 18:44:33.234710 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.234570 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aadef35-91ef-43d9-b10e-18be4abc0110-model-cache" (OuterVolumeSpecName: "model-cache") pod "7aadef35-91ef-43d9-b10e-18be4abc0110" (UID: "7aadef35-91ef-43d9-b10e-18be4abc0110"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:44:33.234710 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.234657 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aadef35-91ef-43d9-b10e-18be4abc0110-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7aadef35-91ef-43d9-b10e-18be4abc0110" (UID: "7aadef35-91ef-43d9-b10e-18be4abc0110"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:44:33.234954 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.234752 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aadef35-91ef-43d9-b10e-18be4abc0110-home" (OuterVolumeSpecName: "home") pod "7aadef35-91ef-43d9-b10e-18be4abc0110" (UID: "7aadef35-91ef-43d9-b10e-18be4abc0110"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:44:33.234954 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.234870 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7aadef35-91ef-43d9-b10e-18be4abc0110-model-cache\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:44:33.234954 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.234882 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7aadef35-91ef-43d9-b10e-18be4abc0110-home\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:44:33.234954 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.234891 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aadef35-91ef-43d9-b10e-18be4abc0110-tmp-dir\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:44:33.236942 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.236884 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aadef35-91ef-43d9-b10e-18be4abc0110-kube-api-access-zj97s" (OuterVolumeSpecName: "kube-api-access-zj97s") pod "7aadef35-91ef-43d9-b10e-18be4abc0110" (UID: "7aadef35-91ef-43d9-b10e-18be4abc0110"). InnerVolumeSpecName "kube-api-access-zj97s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:44:33.237207 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.237190 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aadef35-91ef-43d9-b10e-18be4abc0110-dshm" (OuterVolumeSpecName: "dshm") pod "7aadef35-91ef-43d9-b10e-18be4abc0110" (UID: "7aadef35-91ef-43d9-b10e-18be4abc0110"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:44:33.237207 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.237194 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aadef35-91ef-43d9-b10e-18be4abc0110-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7aadef35-91ef-43d9-b10e-18be4abc0110" (UID: "7aadef35-91ef-43d9-b10e-18be4abc0110"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:44:33.335457 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.335398 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7aadef35-91ef-43d9-b10e-18be4abc0110-dshm\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:44:33.335457 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.335450 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7aadef35-91ef-43d9-b10e-18be4abc0110-tls-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:44:33.335457 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.335467 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj97s\" (UniqueName: \"kubernetes.io/projected/7aadef35-91ef-43d9-b10e-18be4abc0110-kube-api-access-zj97s\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:44:33.702084 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.702024 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" podUID="21b53717-fe8f-469d-99e9-577e55477c8e" containerName="main" containerID="cri-o://6c686a702720a671d9d08ea0357c8f593c4f3a8fcebd2185fe1342c826078ba5" gracePeriod=45 Aug 04 18:44:33.800080 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.800046 2572 generic.go:356] "Generic (PLEG): container finished" podID="7aadef35-91ef-43d9-b10e-18be4abc0110" containerID="82163d57103f779676b40a8890152c89d6b56bab55d9e6d716aa469dd17f9c36" exitCode=0 Aug 04 18:44:33.800080 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.800083 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" Aug 04 18:44:33.800316 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.800126 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" event={"ID":"7aadef35-91ef-43d9-b10e-18be4abc0110","Type":"ContainerDied","Data":"82163d57103f779676b40a8890152c89d6b56bab55d9e6d716aa469dd17f9c36"} Aug 04 18:44:33.800316 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.800162 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf" event={"ID":"7aadef35-91ef-43d9-b10e-18be4abc0110","Type":"ContainerDied","Data":"36fb454d668af13bca80f2eed4ed888abeea439c5fe76f59d4dabee832072df9"} Aug 04 18:44:33.800316 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.800178 2572 scope.go:117] "RemoveContainer" containerID="82163d57103f779676b40a8890152c89d6b56bab55d9e6d716aa469dd17f9c36" Aug 04 18:44:33.813727 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.813694 2572 scope.go:117] "RemoveContainer" containerID="82163d57103f779676b40a8890152c89d6b56bab55d9e6d716aa469dd17f9c36" Aug 04 18:44:33.814026 ip-10-0-137-173 kubenswrapper[2572]: E0804 18:44:33.814008 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82163d57103f779676b40a8890152c89d6b56bab55d9e6d716aa469dd17f9c36\": container with ID starting with 82163d57103f779676b40a8890152c89d6b56bab55d9e6d716aa469dd17f9c36 not found: ID does not exist" containerID="82163d57103f779676b40a8890152c89d6b56bab55d9e6d716aa469dd17f9c36" Aug 04 18:44:33.814094 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.814038 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82163d57103f779676b40a8890152c89d6b56bab55d9e6d716aa469dd17f9c36"} err="failed to get container status \"82163d57103f779676b40a8890152c89d6b56bab55d9e6d716aa469dd17f9c36\": rpc error: code = NotFound desc = could not find container \"82163d57103f779676b40a8890152c89d6b56bab55d9e6d716aa469dd17f9c36\": container with ID starting with 82163d57103f779676b40a8890152c89d6b56bab55d9e6d716aa469dd17f9c36 not found: ID does not exist" Aug 04 18:44:33.823565 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.823546 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf"] Aug 04 18:44:33.827811 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.827790 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-5c8685/sub-test-model-5c8685-kserve-5fc7f4cd77-nzrmf"] Aug 04 18:44:33.943587 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:33.943566 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:44:34.041307 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.041225 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kckv\" (UniqueName: \"kubernetes.io/projected/21b53717-fe8f-469d-99e9-577e55477c8e-kube-api-access-7kckv\") pod \"21b53717-fe8f-469d-99e9-577e55477c8e\" (UID: \"21b53717-fe8f-469d-99e9-577e55477c8e\") " Aug 04 18:44:34.041307 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.041294 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b53717-fe8f-469d-99e9-577e55477c8e-model-cache\") pod \"21b53717-fe8f-469d-99e9-577e55477c8e\" (UID: \"21b53717-fe8f-469d-99e9-577e55477c8e\") " Aug 04 18:44:34.041478 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.041378 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b53717-fe8f-469d-99e9-577e55477c8e-home\") pod \"21b53717-fe8f-469d-99e9-577e55477c8e\" (UID: \"21b53717-fe8f-469d-99e9-577e55477c8e\") " Aug 04 18:44:34.041478 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.041422 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b53717-fe8f-469d-99e9-577e55477c8e-tls-certs\") pod \"21b53717-fe8f-469d-99e9-577e55477c8e\" (UID: \"21b53717-fe8f-469d-99e9-577e55477c8e\") " Aug 04 18:44:34.041478 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.041458 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b53717-fe8f-469d-99e9-577e55477c8e-tmp-dir\") pod \"21b53717-fe8f-469d-99e9-577e55477c8e\" (UID: \"21b53717-fe8f-469d-99e9-577e55477c8e\") " Aug 04 18:44:34.041647 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.041494 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b53717-fe8f-469d-99e9-577e55477c8e-dshm\") pod \"21b53717-fe8f-469d-99e9-577e55477c8e\" (UID: \"21b53717-fe8f-469d-99e9-577e55477c8e\") " Aug 04 18:44:34.041647 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.041593 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b53717-fe8f-469d-99e9-577e55477c8e-model-cache" (OuterVolumeSpecName: "model-cache") pod "21b53717-fe8f-469d-99e9-577e55477c8e" (UID: "21b53717-fe8f-469d-99e9-577e55477c8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:44:34.041647 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.041610 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b53717-fe8f-469d-99e9-577e55477c8e-home" (OuterVolumeSpecName: "home") pod "21b53717-fe8f-469d-99e9-577e55477c8e" (UID: "21b53717-fe8f-469d-99e9-577e55477c8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:44:34.041802 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.041703 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b53717-fe8f-469d-99e9-577e55477c8e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "21b53717-fe8f-469d-99e9-577e55477c8e" (UID: "21b53717-fe8f-469d-99e9-577e55477c8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:44:34.041874 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.041852 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21b53717-fe8f-469d-99e9-577e55477c8e-model-cache\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:44:34.041934 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.041878 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21b53717-fe8f-469d-99e9-577e55477c8e-home\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:44:34.041934 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.041891 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21b53717-fe8f-469d-99e9-577e55477c8e-tmp-dir\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:44:34.043449 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.043424 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b53717-fe8f-469d-99e9-577e55477c8e-kube-api-access-7kckv" (OuterVolumeSpecName: "kube-api-access-7kckv") pod "21b53717-fe8f-469d-99e9-577e55477c8e" (UID: "21b53717-fe8f-469d-99e9-577e55477c8e"). InnerVolumeSpecName "kube-api-access-7kckv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:44:34.043570 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.043525 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b53717-fe8f-469d-99e9-577e55477c8e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21b53717-fe8f-469d-99e9-577e55477c8e" (UID: "21b53717-fe8f-469d-99e9-577e55477c8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:44:34.043570 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.043545 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b53717-fe8f-469d-99e9-577e55477c8e-dshm" (OuterVolumeSpecName: "dshm") pod "21b53717-fe8f-469d-99e9-577e55477c8e" (UID: "21b53717-fe8f-469d-99e9-577e55477c8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:44:34.143061 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.143033 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kckv\" (UniqueName: \"kubernetes.io/projected/21b53717-fe8f-469d-99e9-577e55477c8e-kube-api-access-7kckv\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:44:34.143061 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.143061 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b53717-fe8f-469d-99e9-577e55477c8e-tls-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:44:34.143205 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.143073 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21b53717-fe8f-469d-99e9-577e55477c8e-dshm\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:44:34.805888 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.805859 2572 generic.go:356] "Generic (PLEG): container finished" podID="21b53717-fe8f-469d-99e9-577e55477c8e" containerID="6c686a702720a671d9d08ea0357c8f593c4f3a8fcebd2185fe1342c826078ba5" exitCode=0 Aug 04 18:44:34.806085 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.805903 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" Aug 04 18:44:34.806085 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.805952 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" event={"ID":"21b53717-fe8f-469d-99e9-577e55477c8e","Type":"ContainerDied","Data":"6c686a702720a671d9d08ea0357c8f593c4f3a8fcebd2185fe1342c826078ba5"} Aug 04 18:44:34.806085 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.806000 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz" event={"ID":"21b53717-fe8f-469d-99e9-577e55477c8e","Type":"ContainerDied","Data":"0f7666bd5d3821379ae65384e63df6431c21eb4a97032f2011b5164419ae39b3"} Aug 04 18:44:34.806085 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.806023 2572 scope.go:117] "RemoveContainer" containerID="6c686a702720a671d9d08ea0357c8f593c4f3a8fcebd2185fe1342c826078ba5" Aug 04 18:44:34.814893 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.814879 2572 scope.go:117] "RemoveContainer" containerID="6c686a702720a671d9d08ea0357c8f593c4f3a8fcebd2185fe1342c826078ba5" Aug 04 18:44:34.815179 ip-10-0-137-173 kubenswrapper[2572]: E0804 18:44:34.815161 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c686a702720a671d9d08ea0357c8f593c4f3a8fcebd2185fe1342c826078ba5\": container with ID starting with 6c686a702720a671d9d08ea0357c8f593c4f3a8fcebd2185fe1342c826078ba5 not found: ID does not exist" containerID="6c686a702720a671d9d08ea0357c8f593c4f3a8fcebd2185fe1342c826078ba5" Aug 04 18:44:34.815254 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.815185 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c686a702720a671d9d08ea0357c8f593c4f3a8fcebd2185fe1342c826078ba5"} err="failed to get container status \"6c686a702720a671d9d08ea0357c8f593c4f3a8fcebd2185fe1342c826078ba5\": rpc error: code = NotFound desc = could not find container \"6c686a702720a671d9d08ea0357c8f593c4f3a8fcebd2185fe1342c826078ba5\": container with ID starting with 6c686a702720a671d9d08ea0357c8f593c4f3a8fcebd2185fe1342c826078ba5 not found: ID does not exist" Aug 04 18:44:34.821173 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.821151 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aadef35-91ef-43d9-b10e-18be4abc0110" path="/var/lib/kubelet/pods/7aadef35-91ef-43d9-b10e-18be4abc0110/volumes" Aug 04 18:44:34.828102 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.828083 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz"] Aug 04 18:44:34.834542 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:34.834523 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bc18f1/sub-test-model-bc18f1-kserve-5d8d9fd86f-n2nfz"] Aug 04 18:44:36.821086 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:36.821055 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21b53717-fe8f-469d-99e9-577e55477c8e" path="/var/lib/kubelet/pods/21b53717-fe8f-469d-99e9-577e55477c8e/volumes" Aug 04 18:44:39.685842 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:39.685810 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:44:47.928219 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:47.928129 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:44:48.029641 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:44:48.029611 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:00.166970 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.166927 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc18f1-29764485-8qzst"] Aug 04 18:45:00.167398 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.167266 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21b53717-fe8f-469d-99e9-577e55477c8e" containerName="main" Aug 04 18:45:00.167398 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.167276 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="21b53717-fe8f-469d-99e9-577e55477c8e" containerName="main" Aug 04 18:45:00.167398 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.167287 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aadef35-91ef-43d9-b10e-18be4abc0110" containerName="main" Aug 04 18:45:00.167398 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.167293 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aadef35-91ef-43d9-b10e-18be4abc0110" containerName="main" Aug 04 18:45:00.167398 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.167348 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="21b53717-fe8f-469d-99e9-577e55477c8e" containerName="main" Aug 04 18:45:00.167398 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.167360 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aadef35-91ef-43d9-b10e-18be4abc0110" containerName="main" Aug 04 18:45:00.171748 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.171727 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c8685-29764485-xkclk"] Aug 04 18:45:00.171876 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.171843 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc18f1-29764485-8qzst" Aug 04 18:45:00.174771 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.174753 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764485-8h6kz"] Aug 04 18:45:00.174887 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.174875 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c8685-29764485-xkclk" Aug 04 18:45:00.175151 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.175133 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pnbq7\"" Aug 04 18:45:00.177752 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.177735 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764485-8h6kz" Aug 04 18:45:00.183640 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.183620 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc18f1-29764485-8qzst"] Aug 04 18:45:00.185814 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.185474 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc18f1-29764485-8qzst" Aug 04 18:45:00.186540 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.186518 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c8685-29764485-xkclk"] Aug 04 18:45:00.189883 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.188564 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764485-8h6kz"] Aug 04 18:45:00.194121 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.194099 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c8685-29764485-xkclk" Aug 04 18:45:00.226716 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.226692 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764485-8h6kz" Aug 04 18:45:00.348089 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.347753 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc18f1-29764485-8qzst"] Aug 04 18:45:00.350688 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:45:00.350641 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf49fe828_3ed9_4b3f_ae6e_a4e1b8ba9f8a.slice/crio-cc84d3e10770554d509adcc2be14c69a49ae1ede9589e6cedb79275598eabaaf WatchSource:0}: Error finding container cc84d3e10770554d509adcc2be14c69a49ae1ede9589e6cedb79275598eabaaf: Status 404 returned error can't find the container with id cc84d3e10770554d509adcc2be14c69a49ae1ede9589e6cedb79275598eabaaf Aug 04 18:45:00.395046 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.395022 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764485-8h6kz"] Aug 04 18:45:00.397700 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:45:00.397655 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4fe340_35fe_4c8f_af78_45d769926a01.slice/crio-5930e5b4aea7642e43bae4e9f2b7146fdb86d2287eb8dbf49feb35d702fdcf37 WatchSource:0}: Error finding container 5930e5b4aea7642e43bae4e9f2b7146fdb86d2287eb8dbf49feb35d702fdcf37: Status 404 returned error can't find the container with id 5930e5b4aea7642e43bae4e9f2b7146fdb86d2287eb8dbf49feb35d702fdcf37 Aug 04 18:45:00.575433 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.575355 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c8685-29764485-xkclk"] Aug 04 18:45:00.661768 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:45:00.661735 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d01363_e20c_4b00_a039_b418289cbb0f.slice/crio-9a3cf72a52ccd4bdfc97a1e528c76a57e65fb29d8f705d8d8daee355c3956cfb WatchSource:0}: Error finding container 9a3cf72a52ccd4bdfc97a1e528c76a57e65fb29d8f705d8d8daee355c3956cfb: Status 404 returned error can't find the container with id 9a3cf72a52ccd4bdfc97a1e528c76a57e65fb29d8f705d8d8daee355c3956cfb Aug 04 18:45:00.927053 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.927020 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c8685-29764485-xkclk" event={"ID":"82d01363-e20c-4b00-a039-b418289cbb0f","Type":"ContainerStarted","Data":"fe98b7b201ce1b007d7103ee225333e5662ae3eb85a92222faf3b2a536bf2c74"} Aug 04 18:45:00.927171 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.927087 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c8685-29764485-xkclk" event={"ID":"82d01363-e20c-4b00-a039-b418289cbb0f","Type":"ContainerStarted","Data":"9a3cf72a52ccd4bdfc97a1e528c76a57e65fb29d8f705d8d8daee355c3956cfb"} Aug 04 18:45:00.928454 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.928427 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc18f1-29764485-8qzst" event={"ID":"f49fe828-3ed9-4b3f-ae6e-a4e1b8ba9f8a","Type":"ContainerStarted","Data":"ddc7fd424f71d8f3d27044b6eef81130ae8af4ca37bb8cec5451a2e521df8b63"} Aug 04 18:45:00.928562 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.928471 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc18f1-29764485-8qzst" event={"ID":"f49fe828-3ed9-4b3f-ae6e-a4e1b8ba9f8a","Type":"ContainerStarted","Data":"cc84d3e10770554d509adcc2be14c69a49ae1ede9589e6cedb79275598eabaaf"} Aug 04 18:45:00.929837 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.929818 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764485-8h6kz" event={"ID":"ab4fe340-35fe-4c8f-af78-45d769926a01","Type":"ContainerStarted","Data":"2cf7abde4a72d7bebca9f67203a78c03cc5c409da0e7aa48a7c64b8ec2ad39b1"} Aug 04 18:45:00.929918 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.929850 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764485-8h6kz" event={"ID":"ab4fe340-35fe-4c8f-af78-45d769926a01","Type":"ContainerStarted","Data":"5930e5b4aea7642e43bae4e9f2b7146fdb86d2287eb8dbf49feb35d702fdcf37"} Aug 04 18:45:00.952007 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.951935 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c8685-29764485-xkclk" podStartSLOduration=0.951923316 podStartE2EDuration="951.923316ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:45:00.949051512 +0000 UTC m=+3156.639742562" watchObservedRunningTime="2026-08-04 18:45:00.951923316 +0000 UTC m=+3156.642614358" Aug 04 18:45:00.971729 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:00.971695 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc18f1-29764485-8qzst" podStartSLOduration=0.971657753 podStartE2EDuration="971.657753ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:45:00.97136748 +0000 UTC m=+3156.662058531" watchObservedRunningTime="2026-08-04 18:45:00.971657753 +0000 UTC m=+3156.662348804" Aug 04 18:45:01.935908 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:01.935880 2572 generic.go:356] "Generic (PLEG): container finished" podID="82d01363-e20c-4b00-a039-b418289cbb0f" containerID="fe98b7b201ce1b007d7103ee225333e5662ae3eb85a92222faf3b2a536bf2c74" exitCode=0 Aug 04 18:45:01.936308 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:01.935951 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c8685-29764485-xkclk" event={"ID":"82d01363-e20c-4b00-a039-b418289cbb0f","Type":"ContainerDied","Data":"fe98b7b201ce1b007d7103ee225333e5662ae3eb85a92222faf3b2a536bf2c74"} Aug 04 18:45:01.937377 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:01.937352 2572 generic.go:356] "Generic (PLEG): container finished" podID="f49fe828-3ed9-4b3f-ae6e-a4e1b8ba9f8a" containerID="ddc7fd424f71d8f3d27044b6eef81130ae8af4ca37bb8cec5451a2e521df8b63" exitCode=0 Aug 04 18:45:01.937511 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:01.937444 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc18f1-29764485-8qzst" event={"ID":"f49fe828-3ed9-4b3f-ae6e-a4e1b8ba9f8a","Type":"ContainerDied","Data":"ddc7fd424f71d8f3d27044b6eef81130ae8af4ca37bb8cec5451a2e521df8b63"} Aug 04 18:45:01.938833 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:01.938813 2572 generic.go:356] "Generic (PLEG): container finished" podID="ab4fe340-35fe-4c8f-af78-45d769926a01" containerID="2cf7abde4a72d7bebca9f67203a78c03cc5c409da0e7aa48a7c64b8ec2ad39b1" exitCode=0 Aug 04 18:45:01.938901 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:01.938884 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764485-8h6kz" event={"ID":"ab4fe340-35fe-4c8f-af78-45d769926a01","Type":"ContainerDied","Data":"2cf7abde4a72d7bebca9f67203a78c03cc5c409da0e7aa48a7c64b8ec2ad39b1"} Aug 04 18:45:01.955003 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:01.954958 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764485-8h6kz" podStartSLOduration=1.9549408160000001 podStartE2EDuration="1.954940816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:45:00.992542323 +0000 UTC m=+3156.683233374" watchObservedRunningTime="2026-08-04 18:45:01.954940816 +0000 UTC m=+3157.645631867" Aug 04 18:45:03.077199 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:03.077168 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c8685-29764485-xkclk" Aug 04 18:45:03.113126 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:03.113097 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764485-8h6kz" Aug 04 18:45:03.116054 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:03.116035 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc18f1-29764485-8qzst" Aug 04 18:45:03.950981 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:03.950954 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c8685-29764485-xkclk" Aug 04 18:45:03.950981 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:03.950959 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c8685-29764485-xkclk" event={"ID":"82d01363-e20c-4b00-a039-b418289cbb0f","Type":"ContainerDied","Data":"9a3cf72a52ccd4bdfc97a1e528c76a57e65fb29d8f705d8d8daee355c3956cfb"} Aug 04 18:45:03.951196 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:03.951011 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a3cf72a52ccd4bdfc97a1e528c76a57e65fb29d8f705d8d8daee355c3956cfb" Aug 04 18:45:03.952169 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:03.952150 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc18f1-29764485-8qzst" Aug 04 18:45:03.952300 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:03.952175 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc18f1-29764485-8qzst" event={"ID":"f49fe828-3ed9-4b3f-ae6e-a4e1b8ba9f8a","Type":"ContainerDied","Data":"cc84d3e10770554d509adcc2be14c69a49ae1ede9589e6cedb79275598eabaaf"} Aug 04 18:45:03.952300 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:03.952212 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc84d3e10770554d509adcc2be14c69a49ae1ede9589e6cedb79275598eabaaf" Aug 04 18:45:03.953440 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:03.953414 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764485-8h6kz" Aug 04 18:45:03.953440 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:03.953433 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764485-8h6kz" event={"ID":"ab4fe340-35fe-4c8f-af78-45d769926a01","Type":"ContainerDied","Data":"5930e5b4aea7642e43bae4e9f2b7146fdb86d2287eb8dbf49feb35d702fdcf37"} Aug 04 18:45:03.953562 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:03.953469 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5930e5b4aea7642e43bae4e9f2b7146fdb86d2287eb8dbf49feb35d702fdcf37" Aug 04 18:45:06.890814 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:06.890780 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln"] Aug 04 18:45:06.891185 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:06.891123 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f49fe828-3ed9-4b3f-ae6e-a4e1b8ba9f8a" containerName="cleanup" Aug 04 18:45:06.891185 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:06.891139 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f49fe828-3ed9-4b3f-ae6e-a4e1b8ba9f8a" containerName="cleanup" Aug 04 18:45:06.891185 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:06.891151 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82d01363-e20c-4b00-a039-b418289cbb0f" containerName="cleanup" Aug 04 18:45:06.891185 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:06.891157 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="82d01363-e20c-4b00-a039-b418289cbb0f" containerName="cleanup" Aug 04 18:45:06.891185 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:06.891164 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab4fe340-35fe-4c8f-af78-45d769926a01" containerName="cleanup" Aug 04 18:45:06.891185 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:06.891169 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab4fe340-35fe-4c8f-af78-45d769926a01" containerName="cleanup" Aug 04 18:45:06.891372 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:06.891233 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="82d01363-e20c-4b00-a039-b418289cbb0f" containerName="cleanup" Aug 04 18:45:06.891372 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:06.891241 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab4fe340-35fe-4c8f-af78-45d769926a01" containerName="cleanup" Aug 04 18:45:06.891372 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:06.891247 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f49fe828-3ed9-4b3f-ae6e-a4e1b8ba9f8a" containerName="cleanup" Aug 04 18:45:06.894258 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:06.894241 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:45:06.897408 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:06.897387 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc18f1\"/\"openshift-service-ca.crt\"" Aug 04 18:45:06.898337 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:06.898316 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc18f1\"/\"default-dockercfg-mgdn7\"" Aug 04 18:45:06.898425 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:06.898367 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc18f1\"/\"rate-test-model-bc18f1-kserve-self-signed-certs\"" Aug 04 18:45:06.898425 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:06.898386 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc18f1\"/\"kube-root-ca.crt\"" Aug 04 18:45:06.911012 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:06.910982 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln"] Aug 04 18:45:07.024967 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:07.024933 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2f0838b-051e-47de-8640-907d26138b4e-tls-certs\") pod \"rate-test-model-bc18f1-kserve-5db955bdd7-vtlln\" (UID: \"f2f0838b-051e-47de-8640-907d26138b4e\") " pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:45:07.024967 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:07.024972 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2f0838b-051e-47de-8640-907d26138b4e-model-cache\") pod \"rate-test-model-bc18f1-kserve-5db955bdd7-vtlln\" (UID: \"f2f0838b-051e-47de-8640-907d26138b4e\") " pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:45:07.025179 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:07.024990 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2f0838b-051e-47de-8640-907d26138b4e-dshm\") pod \"rate-test-model-bc18f1-kserve-5db955bdd7-vtlln\" (UID: \"f2f0838b-051e-47de-8640-907d26138b4e\") " pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:45:07.025179 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:07.025110 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2f0838b-051e-47de-8640-907d26138b4e-home\") pod \"rate-test-model-bc18f1-kserve-5db955bdd7-vtlln\" (UID: \"f2f0838b-051e-47de-8640-907d26138b4e\") " pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:45:07.025179 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:07.025159 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2f0838b-051e-47de-8640-907d26138b4e-tmp-dir\") pod \"rate-test-model-bc18f1-kserve-5db955bdd7-vtlln\" (UID: \"f2f0838b-051e-47de-8640-907d26138b4e\") " pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:45:07.025179 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:07.025177 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ksjx\" (UniqueName: \"kubernetes.io/projected/f2f0838b-051e-47de-8640-907d26138b4e-kube-api-access-7ksjx\") pod \"rate-test-model-bc18f1-kserve-5db955bdd7-vtlln\" (UID: \"f2f0838b-051e-47de-8640-907d26138b4e\") " pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:45:07.126020 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:07.125989 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2f0838b-051e-47de-8640-907d26138b4e-home\") pod \"rate-test-model-bc18f1-kserve-5db955bdd7-vtlln\" (UID: \"f2f0838b-051e-47de-8640-907d26138b4e\") " pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:45:07.126181 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:07.126030 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2f0838b-051e-47de-8640-907d26138b4e-tmp-dir\") pod \"rate-test-model-bc18f1-kserve-5db955bdd7-vtlln\" (UID: \"f2f0838b-051e-47de-8640-907d26138b4e\") " pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:45:07.126181 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:07.126050 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ksjx\" (UniqueName: \"kubernetes.io/projected/f2f0838b-051e-47de-8640-907d26138b4e-kube-api-access-7ksjx\") pod \"rate-test-model-bc18f1-kserve-5db955bdd7-vtlln\" (UID: \"f2f0838b-051e-47de-8640-907d26138b4e\") " pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:45:07.126181 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:07.126085 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2f0838b-051e-47de-8640-907d26138b4e-tls-certs\") pod \"rate-test-model-bc18f1-kserve-5db955bdd7-vtlln\" (UID: \"f2f0838b-051e-47de-8640-907d26138b4e\") " pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:45:07.126181 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:07.126101 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2f0838b-051e-47de-8640-907d26138b4e-model-cache\") pod \"rate-test-model-bc18f1-kserve-5db955bdd7-vtlln\" (UID: \"f2f0838b-051e-47de-8640-907d26138b4e\") " pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:45:07.126181 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:07.126116 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2f0838b-051e-47de-8640-907d26138b4e-dshm\") pod \"rate-test-model-bc18f1-kserve-5db955bdd7-vtlln\" (UID: \"f2f0838b-051e-47de-8640-907d26138b4e\") " pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:45:07.126486 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:07.126466 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2f0838b-051e-47de-8640-907d26138b4e-home\") pod \"rate-test-model-bc18f1-kserve-5db955bdd7-vtlln\" (UID: \"f2f0838b-051e-47de-8640-907d26138b4e\") " pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:45:07.126552 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:07.126466 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2f0838b-051e-47de-8640-907d26138b4e-tmp-dir\") pod \"rate-test-model-bc18f1-kserve-5db955bdd7-vtlln\" (UID: \"f2f0838b-051e-47de-8640-907d26138b4e\") " pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:45:07.126552 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:07.126540 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2f0838b-051e-47de-8640-907d26138b4e-model-cache\") pod \"rate-test-model-bc18f1-kserve-5db955bdd7-vtlln\" (UID: \"f2f0838b-051e-47de-8640-907d26138b4e\") " pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:45:07.128510 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:07.128487 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2f0838b-051e-47de-8640-907d26138b4e-dshm\") pod \"rate-test-model-bc18f1-kserve-5db955bdd7-vtlln\" (UID: \"f2f0838b-051e-47de-8640-907d26138b4e\") " pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:45:07.128628 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:07.128569 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2f0838b-051e-47de-8640-907d26138b4e-tls-certs\") pod \"rate-test-model-bc18f1-kserve-5db955bdd7-vtlln\" (UID: \"f2f0838b-051e-47de-8640-907d26138b4e\") " pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:45:07.136882 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:07.136857 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ksjx\" (UniqueName: \"kubernetes.io/projected/f2f0838b-051e-47de-8640-907d26138b4e-kube-api-access-7ksjx\") pod \"rate-test-model-bc18f1-kserve-5db955bdd7-vtlln\" (UID: \"f2f0838b-051e-47de-8640-907d26138b4e\") " pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:45:07.203484 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:07.203401 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:45:07.336520 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:07.336460 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln"] Aug 04 18:45:07.338810 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:45:07.338780 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2f0838b_051e_47de_8640_907d26138b4e.slice/crio-50b5bb5a42a5fda71a4b25dfcdb44a674550527be574249a1b39f88ef4fedcf7 WatchSource:0}: Error finding container 50b5bb5a42a5fda71a4b25dfcdb44a674550527be574249a1b39f88ef4fedcf7: Status 404 returned error can't find the container with id 50b5bb5a42a5fda71a4b25dfcdb44a674550527be574249a1b39f88ef4fedcf7 Aug 04 18:45:07.972105 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:07.972068 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" event={"ID":"f2f0838b-051e-47de-8640-907d26138b4e","Type":"ContainerStarted","Data":"3b2bb2821c42d114789ca1d5ebd49b977ea1dc8e4e270336be251d30ed6f0b51"} Aug 04 18:45:07.972105 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:07.972109 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" event={"ID":"f2f0838b-051e-47de-8640-907d26138b4e","Type":"ContainerStarted","Data":"50b5bb5a42a5fda71a4b25dfcdb44a674550527be574249a1b39f88ef4fedcf7"} Aug 04 18:45:07.994399 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:07.994348 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" podStartSLOduration=1.994334877 podStartE2EDuration="1.994334877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:45:07.991878052 +0000 UTC m=+3163.682569102" watchObservedRunningTime="2026-08-04 18:45:07.994334877 +0000 UTC m=+3163.685025928" Aug 04 18:45:08.357727 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:08.357655 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:45:17.203966 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:17.203934 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:45:17.204351 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:17.203979 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:45:17.216367 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:17.216344 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:45:18.026853 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:18.026827 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:45:35.238771 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:45:35.238714 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:46:17.365015 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:17.364941 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln"] Aug 04 18:46:17.613623 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:17.613589 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:46:19.461138 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:19.461109 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:46:32.383871 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:32.383809 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" podUID="f2f0838b-051e-47de-8640-907d26138b4e" containerName="main" containerID="cri-o://3b2bb2821c42d114789ca1d5ebd49b977ea1dc8e4e270336be251d30ed6f0b51" gracePeriod=45 Aug 04 18:46:32.632856 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:32.632831 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:46:32.655001 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:32.654942 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2f0838b-051e-47de-8640-907d26138b4e-model-cache\") pod \"f2f0838b-051e-47de-8640-907d26138b4e\" (UID: \"f2f0838b-051e-47de-8640-907d26138b4e\") " Aug 04 18:46:32.655001 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:32.654973 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2f0838b-051e-47de-8640-907d26138b4e-home\") pod \"f2f0838b-051e-47de-8640-907d26138b4e\" (UID: \"f2f0838b-051e-47de-8640-907d26138b4e\") " Aug 04 18:46:32.655161 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:32.655004 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2f0838b-051e-47de-8640-907d26138b4e-tls-certs\") pod \"f2f0838b-051e-47de-8640-907d26138b4e\" (UID: \"f2f0838b-051e-47de-8640-907d26138b4e\") " Aug 04 18:46:32.655161 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:32.655039 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2f0838b-051e-47de-8640-907d26138b4e-tmp-dir\") pod \"f2f0838b-051e-47de-8640-907d26138b4e\" (UID: \"f2f0838b-051e-47de-8640-907d26138b4e\") " Aug 04 18:46:32.655161 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:32.655060 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ksjx\" (UniqueName: \"kubernetes.io/projected/f2f0838b-051e-47de-8640-907d26138b4e-kube-api-access-7ksjx\") pod \"f2f0838b-051e-47de-8640-907d26138b4e\" (UID: \"f2f0838b-051e-47de-8640-907d26138b4e\") " Aug 04 18:46:32.655161 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:32.655092 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2f0838b-051e-47de-8640-907d26138b4e-dshm\") pod \"f2f0838b-051e-47de-8640-907d26138b4e\" (UID: \"f2f0838b-051e-47de-8640-907d26138b4e\") " Aug 04 18:46:32.655365 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:32.655201 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f0838b-051e-47de-8640-907d26138b4e-home" (OuterVolumeSpecName: "home") pod "f2f0838b-051e-47de-8640-907d26138b4e" (UID: "f2f0838b-051e-47de-8640-907d26138b4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:46:32.655365 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:32.655217 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f0838b-051e-47de-8640-907d26138b4e-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2f0838b-051e-47de-8640-907d26138b4e" (UID: "f2f0838b-051e-47de-8640-907d26138b4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:46:32.655365 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:32.655296 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f0838b-051e-47de-8640-907d26138b4e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2f0838b-051e-47de-8640-907d26138b4e" (UID: "f2f0838b-051e-47de-8640-907d26138b4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:46:32.655509 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:32.655467 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2f0838b-051e-47de-8640-907d26138b4e-model-cache\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:46:32.655509 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:32.655486 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2f0838b-051e-47de-8640-907d26138b4e-home\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:46:32.655509 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:32.655495 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2f0838b-051e-47de-8640-907d26138b4e-tmp-dir\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:46:32.657310 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:32.657287 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2f0838b-051e-47de-8640-907d26138b4e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2f0838b-051e-47de-8640-907d26138b4e" (UID: "f2f0838b-051e-47de-8640-907d26138b4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:46:32.657412 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:32.657287 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f0838b-051e-47de-8640-907d26138b4e-dshm" (OuterVolumeSpecName: "dshm") pod "f2f0838b-051e-47de-8640-907d26138b4e" (UID: "f2f0838b-051e-47de-8640-907d26138b4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:46:32.657412 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:32.657319 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f0838b-051e-47de-8640-907d26138b4e-kube-api-access-7ksjx" (OuterVolumeSpecName: "kube-api-access-7ksjx") pod "f2f0838b-051e-47de-8640-907d26138b4e" (UID: "f2f0838b-051e-47de-8640-907d26138b4e"). InnerVolumeSpecName "kube-api-access-7ksjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:46:32.756565 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:32.756535 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2f0838b-051e-47de-8640-907d26138b4e-tls-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:46:32.756565 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:32.756559 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ksjx\" (UniqueName: \"kubernetes.io/projected/f2f0838b-051e-47de-8640-907d26138b4e-kube-api-access-7ksjx\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:46:32.756565 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:32.756569 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2f0838b-051e-47de-8640-907d26138b4e-dshm\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:46:33.362161 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:33.362123 2572 generic.go:356] "Generic (PLEG): container finished" podID="f2f0838b-051e-47de-8640-907d26138b4e" containerID="3b2bb2821c42d114789ca1d5ebd49b977ea1dc8e4e270336be251d30ed6f0b51" exitCode=0 Aug 04 18:46:33.362367 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:33.362166 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" Aug 04 18:46:33.362367 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:33.362213 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" event={"ID":"f2f0838b-051e-47de-8640-907d26138b4e","Type":"ContainerDied","Data":"3b2bb2821c42d114789ca1d5ebd49b977ea1dc8e4e270336be251d30ed6f0b51"} Aug 04 18:46:33.362367 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:33.362254 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln" event={"ID":"f2f0838b-051e-47de-8640-907d26138b4e","Type":"ContainerDied","Data":"50b5bb5a42a5fda71a4b25dfcdb44a674550527be574249a1b39f88ef4fedcf7"} Aug 04 18:46:33.362367 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:33.362272 2572 scope.go:117] "RemoveContainer" containerID="3b2bb2821c42d114789ca1d5ebd49b977ea1dc8e4e270336be251d30ed6f0b51" Aug 04 18:46:33.370591 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:33.370579 2572 scope.go:117] "RemoveContainer" containerID="3b2bb2821c42d114789ca1d5ebd49b977ea1dc8e4e270336be251d30ed6f0b51" Aug 04 18:46:33.370843 ip-10-0-137-173 kubenswrapper[2572]: E0804 18:46:33.370825 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b2bb2821c42d114789ca1d5ebd49b977ea1dc8e4e270336be251d30ed6f0b51\": container with ID starting with 3b2bb2821c42d114789ca1d5ebd49b977ea1dc8e4e270336be251d30ed6f0b51 not found: ID does not exist" containerID="3b2bb2821c42d114789ca1d5ebd49b977ea1dc8e4e270336be251d30ed6f0b51" Aug 04 18:46:33.370923 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:33.370854 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2bb2821c42d114789ca1d5ebd49b977ea1dc8e4e270336be251d30ed6f0b51"} err="failed to get container status \"3b2bb2821c42d114789ca1d5ebd49b977ea1dc8e4e270336be251d30ed6f0b51\": rpc error: code = NotFound desc = could not find container \"3b2bb2821c42d114789ca1d5ebd49b977ea1dc8e4e270336be251d30ed6f0b51\": container with ID starting with 3b2bb2821c42d114789ca1d5ebd49b977ea1dc8e4e270336be251d30ed6f0b51 not found: ID does not exist" Aug 04 18:46:33.382382 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:33.382362 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln"] Aug 04 18:46:33.387872 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:33.387851 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bc18f1/rate-test-model-bc18f1-kserve-5db955bdd7-vtlln"] Aug 04 18:46:34.822197 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:34.822167 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2f0838b-051e-47de-8640-907d26138b4e" path="/var/lib/kubelet/pods/f2f0838b-051e-47de-8640-907d26138b4e/volumes" Aug 04 18:46:37.419568 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.419511 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:46:37.706956 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.706883 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm"] Aug 04 18:46:37.707226 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.707214 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2f0838b-051e-47de-8640-907d26138b4e" containerName="main" Aug 04 18:46:37.707280 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.707228 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2f0838b-051e-47de-8640-907d26138b4e" containerName="main" Aug 04 18:46:37.707318 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.707286 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2f0838b-051e-47de-8640-907d26138b4e" containerName="main" Aug 04 18:46:37.710151 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.710135 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.712995 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.712979 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6a6d9f-openshift-default-dockercfg-t29qz\"" Aug 04 18:46:37.728115 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.728088 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm"] Aug 04 18:46:37.733847 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.733826 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:46:37.797802 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.797773 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-istio-envoy\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.797978 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.797818 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7111e954-0f50-496c-9b61-a5944f6bade8-istio-podinfo\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.797978 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.797849 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7111e954-0f50-496c-9b61-a5944f6bade8-istio-token\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.797978 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.797905 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-workload-socket\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.797978 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.797929 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-credential-socket\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.797978 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.797958 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k54t7\" (UniqueName: \"kubernetes.io/projected/7111e954-0f50-496c-9b61-a5944f6bade8-kube-api-access-k54t7\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.798153 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.798005 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-workload-certs\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.798153 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.798056 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7111e954-0f50-496c-9b61-a5944f6bade8-istiod-ca-cert\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.798153 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.798111 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-istio-data\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.899389 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.899354 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k54t7\" (UniqueName: \"kubernetes.io/projected/7111e954-0f50-496c-9b61-a5944f6bade8-kube-api-access-k54t7\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.899389 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.899396 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-workload-certs\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.899606 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.899415 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7111e954-0f50-496c-9b61-a5944f6bade8-istiod-ca-cert\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.899606 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.899453 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-istio-data\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.899606 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.899528 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-istio-envoy\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.899606 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.899582 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7111e954-0f50-496c-9b61-a5944f6bade8-istio-podinfo\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.899606 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.899604 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7111e954-0f50-496c-9b61-a5944f6bade8-istio-token\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.899916 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.899639 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-workload-socket\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.899916 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.899691 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-credential-socket\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.900038 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.899962 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-workload-certs\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.900038 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.899988 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-credential-socket\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.900124 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.900106 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-istio-data\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.900255 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.900235 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-workload-socket\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.900324 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.900312 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7111e954-0f50-496c-9b61-a5944f6bade8-istiod-ca-cert\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.902165 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.902138 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-istio-envoy\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.902392 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.902373 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7111e954-0f50-496c-9b61-a5944f6bade8-istio-podinfo\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.909849 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.909827 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7111e954-0f50-496c-9b61-a5944f6bade8-istio-token\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:37.910127 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:37.910105 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k54t7\" (UniqueName: \"kubernetes.io/projected/7111e954-0f50-496c-9b61-a5944f6bade8-kube-api-access-k54t7\") pod \"e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:38.019474 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:38.019411 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:38.152474 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:38.152449 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm"] Aug 04 18:46:38.154270 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:46:38.154243 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7111e954_0f50_496c_9b61_a5944f6bade8.slice/crio-3b0b5763762aefc205a5ab10a5a235aa3db0091788e21390a66c5c287702597b WatchSource:0}: Error finding container 3b0b5763762aefc205a5ab10a5a235aa3db0091788e21390a66c5c287702597b: Status 404 returned error can't find the container with id 3b0b5763762aefc205a5ab10a5a235aa3db0091788e21390a66c5c287702597b Aug 04 18:46:38.156364 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:38.156331 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:46:38.156462 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:38.156416 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:46:38.156527 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:38.156459 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:46:38.307784 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:38.307755 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:46:38.387808 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:38.387773 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" event={"ID":"7111e954-0f50-496c-9b61-a5944f6bade8","Type":"ContainerStarted","Data":"a1fab6740f92f6eff8baaa3f52d48ff4fc730a726eeec5dd286e094f81156b3e"} Aug 04 18:46:38.387976 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:38.387832 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" event={"ID":"7111e954-0f50-496c-9b61-a5944f6bade8","Type":"ContainerStarted","Data":"3b0b5763762aefc205a5ab10a5a235aa3db0091788e21390a66c5c287702597b"} Aug 04 18:46:38.413247 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:38.413187 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" podStartSLOduration=1.413165886 podStartE2EDuration="1.413165886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:46:38.412472423 +0000 UTC m=+3254.103163475" watchObservedRunningTime="2026-08-04 18:46:38.413165886 +0000 UTC m=+3254.103856938" Aug 04 18:46:39.020124 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:39.020087 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:39.025357 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:39.025335 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:39.393004 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:39.392973 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:39.393976 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:39.393958 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:46:40.362295 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:40.362263 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:46:40.467045 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:46:40.467013 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:01.120149 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:01.120108 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:01.226460 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:01.226431 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:18.789475 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:18.789440 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:27.379908 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:27.379873 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:27.540834 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:27.540805 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:28.371355 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:28.371323 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:28.467466 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:28.467431 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:30.925883 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:30.925841 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:40.433836 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:40.433796 2572 patch_prober.go:28] interesting pod/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Aug 04 18:47:40.434226 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:40.433877 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" podUID="7111e954-0f50-496c-9b61-a5944f6bade8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Aug 04 18:47:41.678142 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:41.678045 2572 generic.go:356] "Generic (PLEG): container finished" podID="7111e954-0f50-496c-9b61-a5944f6bade8" containerID="a1fab6740f92f6eff8baaa3f52d48ff4fc730a726eeec5dd286e094f81156b3e" exitCode=0 Aug 04 18:47:41.678142 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:41.678116 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" event={"ID":"7111e954-0f50-496c-9b61-a5944f6bade8","Type":"ContainerDied","Data":"a1fab6740f92f6eff8baaa3f52d48ff4fc730a726eeec5dd286e094f81156b3e"} Aug 04 18:47:41.678588 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:41.678469 2572 scope.go:117] "RemoveContainer" containerID="a1fab6740f92f6eff8baaa3f52d48ff4fc730a726eeec5dd286e094f81156b3e" Aug 04 18:47:41.679531 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:41.679503 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:47:41.679632 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:41.679568 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:47:41.679632 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:41.679599 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:47:42.686958 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:42.686924 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" event={"ID":"7111e954-0f50-496c-9b61-a5944f6bade8","Type":"ContainerStarted","Data":"5952f135f82d1611ebf25b965a63d60eb70a1f9fa7f7ca36ceb90bd321c15a9d"} Aug 04 18:47:43.020429 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:43.020346 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:47:43.020590 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:43.020527 2572 patch_prober.go:28] interesting pod/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Aug 04 18:47:43.020687 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:43.020578 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" podUID="7111e954-0f50-496c-9b61-a5944f6bade8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Aug 04 18:47:44.020155 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:44.020125 2572 patch_prober.go:28] interesting pod/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Aug 04 18:47:44.020534 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:44.020185 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" podUID="7111e954-0f50-496c-9b61-a5944f6bade8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Aug 04 18:47:45.019982 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:45.019903 2572 patch_prober.go:28] interesting pod/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Aug 04 18:47:45.019982 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:45.019957 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" podUID="7111e954-0f50-496c-9b61-a5944f6bade8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Aug 04 18:47:46.020621 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:46.020580 2572 patch_prober.go:28] interesting pod/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Aug 04 18:47:46.021056 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:46.020641 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" podUID="7111e954-0f50-496c-9b61-a5944f6bade8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Aug 04 18:47:46.314206 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:46.314168 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:46.973761 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:46.973725 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:47.020267 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.020236 2572 patch_prober.go:28] interesting pod/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Aug 04 18:47:47.020432 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.020288 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" podUID="7111e954-0f50-496c-9b61-a5944f6bade8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Aug 04 18:47:47.532393 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.532362 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:47.557199 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.557158 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx"] Aug 04 18:47:47.560176 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.560151 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.565415 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.565390 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-071def-openshift-default-dockercfg-sb7xc\"" Aug 04 18:47:47.614694 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.614636 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-istio-data\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.614895 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.614717 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t52q\" (UniqueName: \"kubernetes.io/projected/1974889b-0585-4331-bcfd-025b966f10dd-kube-api-access-9t52q\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.614895 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.614751 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1974889b-0585-4331-bcfd-025b966f10dd-istiod-ca-cert\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.614895 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.614795 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-istio-envoy\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.614895 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.614827 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1974889b-0585-4331-bcfd-025b966f10dd-istio-token\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.614895 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.614852 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-credential-socket\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.614895 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.614880 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-workload-socket\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.615148 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.614901 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1974889b-0585-4331-bcfd-025b966f10dd-istio-podinfo\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.615148 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.614964 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-workload-certs\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.681404 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.681366 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx"] Aug 04 18:47:47.715454 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.715416 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-workload-certs\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.715640 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.715472 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-istio-data\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.715640 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.715510 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t52q\" (UniqueName: \"kubernetes.io/projected/1974889b-0585-4331-bcfd-025b966f10dd-kube-api-access-9t52q\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.715640 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.715540 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1974889b-0585-4331-bcfd-025b966f10dd-istiod-ca-cert\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.715640 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.715585 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-istio-envoy\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.715640 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.715617 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1974889b-0585-4331-bcfd-025b966f10dd-istio-token\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.715954 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.715646 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-credential-socket\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.715954 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.715708 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-workload-socket\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.715954 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.715732 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1974889b-0585-4331-bcfd-025b966f10dd-istio-podinfo\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.715954 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.715911 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-workload-certs\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.715954 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.715942 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-istio-data\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.716140 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.716092 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-workload-socket\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.716140 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.716126 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-credential-socket\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.716312 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.716294 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1974889b-0585-4331-bcfd-025b966f10dd-istiod-ca-cert\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.718159 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.718137 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-istio-envoy\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.718284 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.718268 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1974889b-0585-4331-bcfd-025b966f10dd-istio-podinfo\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.794813 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.794787 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1974889b-0585-4331-bcfd-025b966f10dd-istio-token\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.795059 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.795038 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t52q\" (UniqueName: \"kubernetes.io/projected/1974889b-0585-4331-bcfd-025b966f10dd-kube-api-access-9t52q\") pod \"e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:47.804154 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.804123 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:47.871351 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:47.871320 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:48.023391 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:48.023359 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:47:48.023626 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:48.023608 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:47:48.024519 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:48.024497 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:47:48.028155 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:48.028133 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx"] Aug 04 18:47:48.030185 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:47:48.030160 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1974889b_0585_4331_bcfd_025b966f10dd.slice/crio-8d74481230b798e1914d25b694c4ddf7100b40dd17482f9e124d058744fa2aee WatchSource:0}: Error finding container 8d74481230b798e1914d25b694c4ddf7100b40dd17482f9e124d058744fa2aee: Status 404 returned error can't find the container with id 8d74481230b798e1914d25b694c4ddf7100b40dd17482f9e124d058744fa2aee Aug 04 18:47:48.032289 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:48.032264 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:47:48.032355 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:48.032323 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:47:48.032355 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:48.032349 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:47:48.718970 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:48.718916 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" event={"ID":"1974889b-0585-4331-bcfd-025b966f10dd","Type":"ContainerStarted","Data":"ef26607f163f27a558d0fb2a321dfa378cf6e88c9e34df847b0e4aaadede01a0"} Aug 04 18:47:48.719368 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:48.718987 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" event={"ID":"1974889b-0585-4331-bcfd-025b966f10dd","Type":"ContainerStarted","Data":"8d74481230b798e1914d25b694c4ddf7100b40dd17482f9e124d058744fa2aee"} Aug 04 18:47:48.747033 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:48.746986 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" podStartSLOduration=1.746969209 podStartE2EDuration="1.746969209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:47:48.745524871 +0000 UTC m=+3324.436215934" watchObservedRunningTime="2026-08-04 18:47:48.746969209 +0000 UTC m=+3324.437660261" Aug 04 18:47:48.872297 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:48.872263 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:49.877824 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:49.877789 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:50.729507 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:50.729465 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:50.730564 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:50.730544 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:47:50.778550 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:50.778517 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:50.889885 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:50.889856 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:54.165768 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:54.165735 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:54.391516 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:54.391480 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:54.796959 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:54.796931 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:47:54.901825 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:47:54.901787 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:48:05.489108 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:05.489058 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:48:06.551623 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:06.551219 2572 patch_prober.go:28] interesting pod/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": read tcp 10.134.0.2:59788->10.134.0.51:15021: read: connection reset by peer" start-of-body= Aug 04 18:48:06.551623 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:06.551304 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" podUID="1974889b-0585-4331-bcfd-025b966f10dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": read tcp 10.134.0.2:59788->10.134.0.51:15021: read: connection reset by peer" Aug 04 18:48:06.838472 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:06.838391 2572 generic.go:356] "Generic (PLEG): container finished" podID="1974889b-0585-4331-bcfd-025b966f10dd" containerID="ef26607f163f27a558d0fb2a321dfa378cf6e88c9e34df847b0e4aaadede01a0" exitCode=0 Aug 04 18:48:06.838472 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:06.838457 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" event={"ID":"1974889b-0585-4331-bcfd-025b966f10dd","Type":"ContainerDied","Data":"ef26607f163f27a558d0fb2a321dfa378cf6e88c9e34df847b0e4aaadede01a0"} Aug 04 18:48:06.838868 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:06.838855 2572 scope.go:117] "RemoveContainer" containerID="ef26607f163f27a558d0fb2a321dfa378cf6e88c9e34df847b0e4aaadede01a0" Aug 04 18:48:06.839979 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:06.839951 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:48:06.840040 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:06.840027 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:48:06.840090 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:06.840078 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Aug 04 18:48:07.845192 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:07.845162 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" event={"ID":"1974889b-0585-4331-bcfd-025b966f10dd","Type":"ContainerStarted","Data":"d8c9f609054d16b2c92beb6361fdfe34f76da1c76a5da7d2b04ad28fc9ccf8fe"} Aug 04 18:48:07.872411 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:07.872370 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:48:07.872691 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:07.872640 2572 patch_prober.go:28] interesting pod/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Aug 04 18:48:07.872785 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:07.872722 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" podUID="1974889b-0585-4331-bcfd-025b966f10dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Aug 04 18:48:08.871712 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:08.871680 2572 patch_prober.go:28] interesting pod/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Aug 04 18:48:08.872267 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:08.872238 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" podUID="1974889b-0585-4331-bcfd-025b966f10dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Aug 04 18:48:09.872277 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:09.872236 2572 patch_prober.go:28] interesting pod/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Aug 04 18:48:09.872778 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:09.872337 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" podUID="1974889b-0585-4331-bcfd-025b966f10dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Aug 04 18:48:10.872211 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:10.872176 2572 patch_prober.go:28] interesting pod/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Aug 04 18:48:10.872503 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:10.872475 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" podUID="1974889b-0585-4331-bcfd-025b966f10dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Aug 04 18:48:11.873298 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:11.873264 2572 patch_prober.go:28] interesting pod/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Aug 04 18:48:11.873958 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:11.873923 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" podUID="1974889b-0585-4331-bcfd-025b966f10dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Aug 04 18:48:12.874928 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:12.874900 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:48:12.875314 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:12.875148 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:48:12.875798 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:12.875776 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:48:34.091081 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.091052 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck"] Aug 04 18:48:34.093537 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.093521 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:48:34.097807 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.097788 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-6a6d9f\"/\"kube-root-ca.crt\"" Aug 04 18:48:34.097938 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.097834 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-6a6d9f\"/\"ipp-route-6a6d9f-kserve-self-signed-certs\"" Aug 04 18:48:34.097938 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.097916 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-6a6d9f\"/\"openshift-service-ca.crt\"" Aug 04 18:48:34.098057 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.097933 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-6a6d9f\"/\"default-dockercfg-hrscz\"" Aug 04 18:48:34.109687 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.109647 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck"] Aug 04 18:48:34.183157 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.183125 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:48:34.235035 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.235006 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hr8t\" (UniqueName: \"kubernetes.io/projected/b25560a0-9b78-44b1-a363-6fcc674f12bf-kube-api-access-9hr8t\") pod \"ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck\" (UID: \"b25560a0-9b78-44b1-a363-6fcc674f12bf\") " pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:48:34.235167 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.235048 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b25560a0-9b78-44b1-a363-6fcc674f12bf-home\") pod \"ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck\" (UID: \"b25560a0-9b78-44b1-a363-6fcc674f12bf\") " pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:48:34.235167 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.235066 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b25560a0-9b78-44b1-a363-6fcc674f12bf-tmp-dir\") pod \"ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck\" (UID: \"b25560a0-9b78-44b1-a363-6fcc674f12bf\") " pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:48:34.235167 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.235128 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b25560a0-9b78-44b1-a363-6fcc674f12bf-tls-certs\") pod \"ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck\" (UID: \"b25560a0-9b78-44b1-a363-6fcc674f12bf\") " pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:48:34.235273 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.235168 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b25560a0-9b78-44b1-a363-6fcc674f12bf-model-cache\") pod \"ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck\" (UID: \"b25560a0-9b78-44b1-a363-6fcc674f12bf\") " pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:48:34.235273 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.235187 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b25560a0-9b78-44b1-a363-6fcc674f12bf-dshm\") pod \"ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck\" (UID: \"b25560a0-9b78-44b1-a363-6fcc674f12bf\") " pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:48:34.336624 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.336594 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b25560a0-9b78-44b1-a363-6fcc674f12bf-home\") pod \"ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck\" (UID: \"b25560a0-9b78-44b1-a363-6fcc674f12bf\") " pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:48:34.336624 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.336627 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b25560a0-9b78-44b1-a363-6fcc674f12bf-tmp-dir\") pod \"ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck\" (UID: \"b25560a0-9b78-44b1-a363-6fcc674f12bf\") " pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:48:34.336837 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.336653 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b25560a0-9b78-44b1-a363-6fcc674f12bf-tls-certs\") pod \"ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck\" (UID: \"b25560a0-9b78-44b1-a363-6fcc674f12bf\") " pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:48:34.336837 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.336701 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b25560a0-9b78-44b1-a363-6fcc674f12bf-model-cache\") pod \"ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck\" (UID: \"b25560a0-9b78-44b1-a363-6fcc674f12bf\") " pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:48:34.336837 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.336723 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b25560a0-9b78-44b1-a363-6fcc674f12bf-dshm\") pod \"ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck\" (UID: \"b25560a0-9b78-44b1-a363-6fcc674f12bf\") " pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:48:34.336837 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.336794 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hr8t\" (UniqueName: \"kubernetes.io/projected/b25560a0-9b78-44b1-a363-6fcc674f12bf-kube-api-access-9hr8t\") pod \"ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck\" (UID: \"b25560a0-9b78-44b1-a363-6fcc674f12bf\") " pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:48:34.337084 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.337062 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b25560a0-9b78-44b1-a363-6fcc674f12bf-tmp-dir\") pod \"ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck\" (UID: \"b25560a0-9b78-44b1-a363-6fcc674f12bf\") " pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:48:34.337157 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.337075 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b25560a0-9b78-44b1-a363-6fcc674f12bf-home\") pod \"ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck\" (UID: \"b25560a0-9b78-44b1-a363-6fcc674f12bf\") " pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:48:34.337157 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.337150 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b25560a0-9b78-44b1-a363-6fcc674f12bf-model-cache\") pod \"ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck\" (UID: \"b25560a0-9b78-44b1-a363-6fcc674f12bf\") " pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:48:34.339040 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.339018 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b25560a0-9b78-44b1-a363-6fcc674f12bf-dshm\") pod \"ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck\" (UID: \"b25560a0-9b78-44b1-a363-6fcc674f12bf\") " pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:48:34.339304 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.339288 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b25560a0-9b78-44b1-a363-6fcc674f12bf-tls-certs\") pod \"ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck\" (UID: \"b25560a0-9b78-44b1-a363-6fcc674f12bf\") " pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:48:34.358243 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.358188 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hr8t\" (UniqueName: \"kubernetes.io/projected/b25560a0-9b78-44b1-a363-6fcc674f12bf-kube-api-access-9hr8t\") pod \"ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck\" (UID: \"b25560a0-9b78-44b1-a363-6fcc674f12bf\") " pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:48:34.403136 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.403097 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:48:34.570927 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.570905 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck"] Aug 04 18:48:34.572845 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:48:34.572812 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb25560a0_9b78_44b1_a363_6fcc674f12bf.slice/crio-04934412f98cbcf01e3da472afc36701a13b9f6e137af62894cfc571a4f23e77 WatchSource:0}: Error finding container 04934412f98cbcf01e3da472afc36701a13b9f6e137af62894cfc571a4f23e77: Status 404 returned error can't find the container with id 04934412f98cbcf01e3da472afc36701a13b9f6e137af62894cfc571a4f23e77 Aug 04 18:48:34.983967 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.983876 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" event={"ID":"b25560a0-9b78-44b1-a363-6fcc674f12bf","Type":"ContainerStarted","Data":"2612d464f9c04961bbecf5caa80a119756dc6e8c124e3431a51e149cdd81c683"} Aug 04 18:48:34.983967 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:34.983921 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" event={"ID":"b25560a0-9b78-44b1-a363-6fcc674f12bf","Type":"ContainerStarted","Data":"04934412f98cbcf01e3da472afc36701a13b9f6e137af62894cfc571a4f23e77"} Aug 04 18:48:35.004295 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:35.004245 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" podStartSLOduration=1.004230804 podStartE2EDuration="1.004230804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:48:35.001960951 +0000 UTC m=+3370.692652015" watchObservedRunningTime="2026-08-04 18:48:35.004230804 +0000 UTC m=+3370.694921855" Aug 04 18:48:44.403562 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:44.403526 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:48:44.403562 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:44.403572 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:48:44.416137 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:44.416111 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:48:45.040559 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:48:45.040528 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:49:13.577797 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:13.577756 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:13.963091 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:13.963007 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:14.290539 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:14.290450 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:16.714730 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:16.714627 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:16.812324 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:16.812288 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:21.200602 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:21.200539 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:21.301507 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:21.301473 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:21.955538 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:21.955498 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:22.112679 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:22.112619 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:36.615607 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:36.615573 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:37.373751 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:37.373714 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f2f60a-66xkm"] Aug 04 18:49:37.376292 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:37.376273 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f2f60a-66xkm" Aug 04 18:49:37.379104 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:37.379054 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 18:49:37.379334 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:37.379317 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pnbq7\"" Aug 04 18:49:37.388641 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:37.388618 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f2f60a-66xkm"] Aug 04 18:49:37.505426 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:37.505397 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f5f00a-0a85-41a4-aeae-85065dc54145-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f2f60a-66xkm\" (UID: \"75f5f00a-0a85-41a4-aeae-85065dc54145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f2f60a-66xkm" Aug 04 18:49:37.606740 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:37.606706 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f5f00a-0a85-41a4-aeae-85065dc54145-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f2f60a-66xkm\" (UID: \"75f5f00a-0a85-41a4-aeae-85065dc54145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f2f60a-66xkm" Aug 04 18:49:37.607313 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:37.607295 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f5f00a-0a85-41a4-aeae-85065dc54145-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f2f60a-66xkm\" (UID: \"75f5f00a-0a85-41a4-aeae-85065dc54145\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f2f60a-66xkm" Aug 04 18:49:37.686507 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:37.686432 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f2f60a-66xkm" Aug 04 18:49:37.841200 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:37.841173 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f2f60a-66xkm"] Aug 04 18:49:37.843432 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:49:37.843403 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f5f00a_0a85_41a4_aeae_85065dc54145.slice/crio-e7dcc9500d13f65d9602dc6d577f8e74b642c6f96515d4d9f2689a9408473293 WatchSource:0}: Error finding container e7dcc9500d13f65d9602dc6d577f8e74b642c6f96515d4d9f2689a9408473293: Status 404 returned error can't find the container with id e7dcc9500d13f65d9602dc6d577f8e74b642c6f96515d4d9f2689a9408473293 Aug 04 18:49:38.278179 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:38.278147 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f2f60a-66xkm" event={"ID":"75f5f00a-0a85-41a4-aeae-85065dc54145","Type":"ContainerStarted","Data":"2f289f9313fbfa64e472a25d0cbe98d7fa2c3949c435a9f82ecbbcaaf2ab89f6"} Aug 04 18:49:38.278350 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:38.278205 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f2f60a-66xkm" event={"ID":"75f5f00a-0a85-41a4-aeae-85065dc54145","Type":"ContainerStarted","Data":"e7dcc9500d13f65d9602dc6d577f8e74b642c6f96515d4d9f2689a9408473293"} Aug 04 18:49:38.306403 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:38.306355 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f2f60a-66xkm" podStartSLOduration=1.306338572 podStartE2EDuration="1.306338572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:49:38.305176678 +0000 UTC m=+3433.995867728" watchObservedRunningTime="2026-08-04 18:49:38.306338572 +0000 UTC m=+3433.997029622" Aug 04 18:49:39.285034 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:39.285002 2572 generic.go:356] "Generic (PLEG): container finished" podID="75f5f00a-0a85-41a4-aeae-85065dc54145" containerID="2f289f9313fbfa64e472a25d0cbe98d7fa2c3949c435a9f82ecbbcaaf2ab89f6" exitCode=0 Aug 04 18:49:39.285468 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:39.285055 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f2f60a-66xkm" event={"ID":"75f5f00a-0a85-41a4-aeae-85065dc54145","Type":"ContainerDied","Data":"2f289f9313fbfa64e472a25d0cbe98d7fa2c3949c435a9f82ecbbcaaf2ab89f6"} Aug 04 18:49:40.416678 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:40.416647 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f2f60a-66xkm" Aug 04 18:49:40.530494 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:40.530467 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f5f00a-0a85-41a4-aeae-85065dc54145-maas-api-service-ca\") pod \"75f5f00a-0a85-41a4-aeae-85065dc54145\" (UID: \"75f5f00a-0a85-41a4-aeae-85065dc54145\") " Aug 04 18:49:40.530856 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:40.530834 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f5f00a-0a85-41a4-aeae-85065dc54145-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "75f5f00a-0a85-41a4-aeae-85065dc54145" (UID: "75f5f00a-0a85-41a4-aeae-85065dc54145"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:49:40.632161 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:40.632130 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f5f00a-0a85-41a4-aeae-85065dc54145-maas-api-service-ca\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:49:41.296579 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:41.296548 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f2f60a-66xkm" Aug 04 18:49:41.296809 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:41.296588 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f2f60a-66xkm" event={"ID":"75f5f00a-0a85-41a4-aeae-85065dc54145","Type":"ContainerDied","Data":"e7dcc9500d13f65d9602dc6d577f8e74b642c6f96515d4d9f2689a9408473293"} Aug 04 18:49:41.296809 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:41.296635 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7dcc9500d13f65d9602dc6d577f8e74b642c6f96515d4d9f2689a9408473293" Aug 04 18:49:49.307876 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:49.307837 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:49.406909 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:49.406818 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:49.480620 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:49.480568 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:49.559612 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:49.559584 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:49.564054 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:49.564022 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f2f60a-66xkm"] Aug 04 18:49:49.573203 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:49.569373 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f2f60a-66xkm"] Aug 04 18:49:49.960437 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:49.960398 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:50.821741 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:50.821705 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75f5f00a-0a85-41a4-aeae-85065dc54145" path="/var/lib/kubelet/pods/75f5f00a-0a85-41a4-aeae-85065dc54145/volumes" Aug 04 18:49:57.049165 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:57.049132 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:57.129905 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:57.129872 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:57.368865 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:57.368765 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:49:57.626911 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:49:57.626825 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:50:06.661102 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:06.661054 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:50:07.286869 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:07.286824 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:50:20.420399 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:20.420344 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:50:20.551158 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:20.551122 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:50:20.697334 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:20.697243 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:50:21.398088 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:21.398057 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:50:21.499589 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:21.499557 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:50:26.187257 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:26.187223 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck"] Aug 04 18:50:26.244091 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:26.244057 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:50:41.206794 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.206734 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" podUID="b25560a0-9b78-44b1-a363-6fcc674f12bf" containerName="main" containerID="cri-o://2612d464f9c04961bbecf5caa80a119756dc6e8c124e3431a51e149cdd81c683" gracePeriod=45 Aug 04 18:50:41.448704 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.448681 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:50:41.567019 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.566991 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b25560a0-9b78-44b1-a363-6fcc674f12bf-tls-certs\") pod \"b25560a0-9b78-44b1-a363-6fcc674f12bf\" (UID: \"b25560a0-9b78-44b1-a363-6fcc674f12bf\") " Aug 04 18:50:41.567192 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.567029 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hr8t\" (UniqueName: \"kubernetes.io/projected/b25560a0-9b78-44b1-a363-6fcc674f12bf-kube-api-access-9hr8t\") pod \"b25560a0-9b78-44b1-a363-6fcc674f12bf\" (UID: \"b25560a0-9b78-44b1-a363-6fcc674f12bf\") " Aug 04 18:50:41.567192 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.567047 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b25560a0-9b78-44b1-a363-6fcc674f12bf-dshm\") pod \"b25560a0-9b78-44b1-a363-6fcc674f12bf\" (UID: \"b25560a0-9b78-44b1-a363-6fcc674f12bf\") " Aug 04 18:50:41.567192 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.567073 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b25560a0-9b78-44b1-a363-6fcc674f12bf-model-cache\") pod \"b25560a0-9b78-44b1-a363-6fcc674f12bf\" (UID: \"b25560a0-9b78-44b1-a363-6fcc674f12bf\") " Aug 04 18:50:41.567192 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.567125 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b25560a0-9b78-44b1-a363-6fcc674f12bf-home\") pod \"b25560a0-9b78-44b1-a363-6fcc674f12bf\" (UID: \"b25560a0-9b78-44b1-a363-6fcc674f12bf\") " Aug 04 18:50:41.567192 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.567192 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b25560a0-9b78-44b1-a363-6fcc674f12bf-tmp-dir\") pod \"b25560a0-9b78-44b1-a363-6fcc674f12bf\" (UID: \"b25560a0-9b78-44b1-a363-6fcc674f12bf\") " Aug 04 18:50:41.567473 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.567404 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25560a0-9b78-44b1-a363-6fcc674f12bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "b25560a0-9b78-44b1-a363-6fcc674f12bf" (UID: "b25560a0-9b78-44b1-a363-6fcc674f12bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:50:41.567473 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.567419 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25560a0-9b78-44b1-a363-6fcc674f12bf-home" (OuterVolumeSpecName: "home") pod "b25560a0-9b78-44b1-a363-6fcc674f12bf" (UID: "b25560a0-9b78-44b1-a363-6fcc674f12bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:50:41.567583 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.567556 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25560a0-9b78-44b1-a363-6fcc674f12bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b25560a0-9b78-44b1-a363-6fcc674f12bf" (UID: "b25560a0-9b78-44b1-a363-6fcc674f12bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:50:41.567583 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.567578 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b25560a0-9b78-44b1-a363-6fcc674f12bf-model-cache\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:50:41.567817 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.567592 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b25560a0-9b78-44b1-a363-6fcc674f12bf-home\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:50:41.569456 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.569426 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25560a0-9b78-44b1-a363-6fcc674f12bf-kube-api-access-9hr8t" (OuterVolumeSpecName: "kube-api-access-9hr8t") pod "b25560a0-9b78-44b1-a363-6fcc674f12bf" (UID: "b25560a0-9b78-44b1-a363-6fcc674f12bf"). InnerVolumeSpecName "kube-api-access-9hr8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:50:41.569456 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.569435 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25560a0-9b78-44b1-a363-6fcc674f12bf-dshm" (OuterVolumeSpecName: "dshm") pod "b25560a0-9b78-44b1-a363-6fcc674f12bf" (UID: "b25560a0-9b78-44b1-a363-6fcc674f12bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:50:41.569606 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.569475 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b25560a0-9b78-44b1-a363-6fcc674f12bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b25560a0-9b78-44b1-a363-6fcc674f12bf" (UID: "b25560a0-9b78-44b1-a363-6fcc674f12bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 18:50:41.587977 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.587948 2572 generic.go:356] "Generic (PLEG): container finished" podID="b25560a0-9b78-44b1-a363-6fcc674f12bf" containerID="2612d464f9c04961bbecf5caa80a119756dc6e8c124e3431a51e149cdd81c683" exitCode=0 Aug 04 18:50:41.588105 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.587985 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" event={"ID":"b25560a0-9b78-44b1-a363-6fcc674f12bf","Type":"ContainerDied","Data":"2612d464f9c04961bbecf5caa80a119756dc6e8c124e3431a51e149cdd81c683"} Aug 04 18:50:41.588105 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.587990 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" Aug 04 18:50:41.588105 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.588012 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck" event={"ID":"b25560a0-9b78-44b1-a363-6fcc674f12bf","Type":"ContainerDied","Data":"04934412f98cbcf01e3da472afc36701a13b9f6e137af62894cfc571a4f23e77"} Aug 04 18:50:41.588105 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.588027 2572 scope.go:117] "RemoveContainer" containerID="2612d464f9c04961bbecf5caa80a119756dc6e8c124e3431a51e149cdd81c683" Aug 04 18:50:41.596801 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.596782 2572 scope.go:117] "RemoveContainer" containerID="2612d464f9c04961bbecf5caa80a119756dc6e8c124e3431a51e149cdd81c683" Aug 04 18:50:41.597046 ip-10-0-137-173 kubenswrapper[2572]: E0804 18:50:41.597027 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2612d464f9c04961bbecf5caa80a119756dc6e8c124e3431a51e149cdd81c683\": container with ID starting with 2612d464f9c04961bbecf5caa80a119756dc6e8c124e3431a51e149cdd81c683 not found: ID does not exist" containerID="2612d464f9c04961bbecf5caa80a119756dc6e8c124e3431a51e149cdd81c683" Aug 04 18:50:41.597099 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.597054 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2612d464f9c04961bbecf5caa80a119756dc6e8c124e3431a51e149cdd81c683"} err="failed to get container status \"2612d464f9c04961bbecf5caa80a119756dc6e8c124e3431a51e149cdd81c683\": rpc error: code = NotFound desc = could not find container \"2612d464f9c04961bbecf5caa80a119756dc6e8c124e3431a51e149cdd81c683\": container with ID starting with 2612d464f9c04961bbecf5caa80a119756dc6e8c124e3431a51e149cdd81c683 not found: ID does not exist" Aug 04 18:50:41.614056 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.614030 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck"] Aug 04 18:50:41.620455 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.620436 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-6a6d9f/ipp-route-6a6d9f-kserve-84b8cdf6b7-g88ck"] Aug 04 18:50:41.668182 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.668157 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b25560a0-9b78-44b1-a363-6fcc674f12bf-tmp-dir\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:50:41.668182 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.668179 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b25560a0-9b78-44b1-a363-6fcc674f12bf-tls-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:50:41.668314 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.668189 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hr8t\" (UniqueName: \"kubernetes.io/projected/b25560a0-9b78-44b1-a363-6fcc674f12bf-kube-api-access-9hr8t\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:50:41.668314 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:41.668198 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b25560a0-9b78-44b1-a363-6fcc674f12bf-dshm\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:50:42.821788 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:50:42.821755 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b25560a0-9b78-44b1-a363-6fcc674f12bf" path="/var/lib/kubelet/pods/b25560a0-9b78-44b1-a363-6fcc674f12bf/volumes" Aug 04 18:51:03.024936 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:03.024869 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:04.744617 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:04.744584 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:11.461922 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:11.461882 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:11.579258 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:11.579207 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:11.733373 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:11.733296 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm"] Aug 04 18:51:11.733537 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:11.733509 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" podUID="7111e954-0f50-496c-9b61-a5944f6bade8" containerName="istio-proxy" containerID="cri-o://5952f135f82d1611ebf25b965a63d60eb70a1f9fa7f7ca36ceb90bd321c15a9d" gracePeriod=30 Aug 04 18:51:11.761910 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:11.761882 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:11.996105 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:11.996025 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:12.154477 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:12.154443 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-071def-fnsp7"] Aug 04 18:51:12.154833 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:12.154820 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b25560a0-9b78-44b1-a363-6fcc674f12bf" containerName="main" Aug 04 18:51:12.154881 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:12.154835 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b25560a0-9b78-44b1-a363-6fcc674f12bf" containerName="main" Aug 04 18:51:12.154881 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:12.154851 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f5f00a-0a85-41a4-aeae-85065dc54145" containerName="revoke-keys" Aug 04 18:51:12.154881 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:12.154856 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f5f00a-0a85-41a4-aeae-85065dc54145" containerName="revoke-keys" Aug 04 18:51:12.154977 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:12.154907 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="75f5f00a-0a85-41a4-aeae-85065dc54145" containerName="revoke-keys" Aug 04 18:51:12.154977 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:12.154916 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b25560a0-9b78-44b1-a363-6fcc674f12bf" containerName="main" Aug 04 18:51:12.157620 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:12.157605 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-071def-fnsp7" Aug 04 18:51:12.159941 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:12.159923 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 18:51:12.160044 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:12.159981 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pnbq7\"" Aug 04 18:51:12.165498 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:12.165464 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-071def-fnsp7"] Aug 04 18:51:12.230315 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:12.230282 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fa775b-b7d7-4ea4-9e34-5a77d2266434-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-071def-fnsp7\" (UID: \"c4fa775b-b7d7-4ea4-9e34-5a77d2266434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-071def-fnsp7" Aug 04 18:51:12.331651 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:12.331617 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fa775b-b7d7-4ea4-9e34-5a77d2266434-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-071def-fnsp7\" (UID: \"c4fa775b-b7d7-4ea4-9e34-5a77d2266434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-071def-fnsp7" Aug 04 18:51:12.332259 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:12.332239 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fa775b-b7d7-4ea4-9e34-5a77d2266434-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-071def-fnsp7\" (UID: \"c4fa775b-b7d7-4ea4-9e34-5a77d2266434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-071def-fnsp7" Aug 04 18:51:12.469857 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:12.469824 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-071def-fnsp7" Aug 04 18:51:12.610842 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:12.610822 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-071def-fnsp7"] Aug 04 18:51:12.613235 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:51:12.613198 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4fa775b_b7d7_4ea4_9e34_5a77d2266434.slice/crio-ba6478a73cf08f7a0ef1a12c26b235268e433480ba3466ad73ff4fe5a6336c7c WatchSource:0}: Error finding container ba6478a73cf08f7a0ef1a12c26b235268e433480ba3466ad73ff4fe5a6336c7c: Status 404 returned error can't find the container with id ba6478a73cf08f7a0ef1a12c26b235268e433480ba3466ad73ff4fe5a6336c7c Aug 04 18:51:12.740973 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:12.740944 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-071def-fnsp7" event={"ID":"c4fa775b-b7d7-4ea4-9e34-5a77d2266434","Type":"ContainerStarted","Data":"ba6478a73cf08f7a0ef1a12c26b235268e433480ba3466ad73ff4fe5a6336c7c"} Aug 04 18:51:13.533278 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:13.533177 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:13.632921 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:13.632889 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:13.746856 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:13.746829 2572 generic.go:356] "Generic (PLEG): container finished" podID="c4fa775b-b7d7-4ea4-9e34-5a77d2266434" containerID="0849c9264a500788835496eb38f7f569b0bf47e118a5a3be0ed52d9ab020db94" exitCode=0 Aug 04 18:51:13.747015 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:13.746864 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-071def-fnsp7" event={"ID":"c4fa775b-b7d7-4ea4-9e34-5a77d2266434","Type":"ContainerDied","Data":"0849c9264a500788835496eb38f7f569b0bf47e118a5a3be0ed52d9ab020db94"} Aug 04 18:51:14.885511 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:14.885484 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-071def-fnsp7" Aug 04 18:51:14.955895 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:14.955868 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fa775b-b7d7-4ea4-9e34-5a77d2266434-maas-api-service-ca\") pod \"c4fa775b-b7d7-4ea4-9e34-5a77d2266434\" (UID: \"c4fa775b-b7d7-4ea4-9e34-5a77d2266434\") " Aug 04 18:51:14.956194 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:14.956172 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4fa775b-b7d7-4ea4-9e34-5a77d2266434-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c4fa775b-b7d7-4ea4-9e34-5a77d2266434" (UID: "c4fa775b-b7d7-4ea4-9e34-5a77d2266434"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:51:15.057380 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:15.057351 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4fa775b-b7d7-4ea4-9e34-5a77d2266434-maas-api-service-ca\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:15.757300 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:15.757269 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-071def-fnsp7" event={"ID":"c4fa775b-b7d7-4ea4-9e34-5a77d2266434","Type":"ContainerDied","Data":"ba6478a73cf08f7a0ef1a12c26b235268e433480ba3466ad73ff4fe5a6336c7c"} Aug 04 18:51:15.757476 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:15.757317 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba6478a73cf08f7a0ef1a12c26b235268e433480ba3466ad73ff4fe5a6336c7c" Aug 04 18:51:15.757476 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:15.757277 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-071def-fnsp7" Aug 04 18:51:16.978703 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:16.978679 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:51:17.076088 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.076055 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-istio-envoy\") pod \"7111e954-0f50-496c-9b61-a5944f6bade8\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " Aug 04 18:51:17.076262 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.076095 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7111e954-0f50-496c-9b61-a5944f6bade8-istio-token\") pod \"7111e954-0f50-496c-9b61-a5944f6bade8\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " Aug 04 18:51:17.076262 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.076123 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7111e954-0f50-496c-9b61-a5944f6bade8-istio-podinfo\") pod \"7111e954-0f50-496c-9b61-a5944f6bade8\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " Aug 04 18:51:17.076262 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.076144 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-workload-socket\") pod \"7111e954-0f50-496c-9b61-a5944f6bade8\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " Aug 04 18:51:17.076424 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.076339 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-istio-data\") pod \"7111e954-0f50-496c-9b61-a5944f6bade8\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " Aug 04 18:51:17.076424 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.076379 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7111e954-0f50-496c-9b61-a5944f6bade8" (UID: "7111e954-0f50-496c-9b61-a5944f6bade8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:51:17.076424 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.076401 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7111e954-0f50-496c-9b61-a5944f6bade8-istiod-ca-cert\") pod \"7111e954-0f50-496c-9b61-a5944f6bade8\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " Aug 04 18:51:17.076577 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.076448 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-workload-certs\") pod \"7111e954-0f50-496c-9b61-a5944f6bade8\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " Aug 04 18:51:17.076577 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.076509 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-credential-socket\") pod \"7111e954-0f50-496c-9b61-a5944f6bade8\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " Aug 04 18:51:17.076577 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.076541 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k54t7\" (UniqueName: \"kubernetes.io/projected/7111e954-0f50-496c-9b61-a5944f6bade8-kube-api-access-k54t7\") pod \"7111e954-0f50-496c-9b61-a5944f6bade8\" (UID: \"7111e954-0f50-496c-9b61-a5944f6bade8\") " Aug 04 18:51:17.076761 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.076568 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-istio-data" (OuterVolumeSpecName: "istio-data") pod "7111e954-0f50-496c-9b61-a5944f6bade8" (UID: "7111e954-0f50-496c-9b61-a5944f6bade8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:51:17.076761 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.076687 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7111e954-0f50-496c-9b61-a5944f6bade8" (UID: "7111e954-0f50-496c-9b61-a5944f6bade8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:51:17.076853 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.076754 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7111e954-0f50-496c-9b61-a5944f6bade8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7111e954-0f50-496c-9b61-a5944f6bade8" (UID: "7111e954-0f50-496c-9b61-a5944f6bade8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:51:17.076853 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.076799 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7111e954-0f50-496c-9b61-a5944f6bade8" (UID: "7111e954-0f50-496c-9b61-a5944f6bade8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:51:17.076962 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.076942 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-credential-socket\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:17.077004 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.076970 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-workload-socket\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:17.077004 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.076986 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-istio-data\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:17.077004 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.076998 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7111e954-0f50-496c-9b61-a5944f6bade8-istiod-ca-cert\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:17.077150 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.077010 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-workload-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:17.078731 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.078701 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7111e954-0f50-496c-9b61-a5944f6bade8" (UID: "7111e954-0f50-496c-9b61-a5944f6bade8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:51:17.078861 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.078774 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7111e954-0f50-496c-9b61-a5944f6bade8-kube-api-access-k54t7" (OuterVolumeSpecName: "kube-api-access-k54t7") pod "7111e954-0f50-496c-9b61-a5944f6bade8" (UID: "7111e954-0f50-496c-9b61-a5944f6bade8"). InnerVolumeSpecName "kube-api-access-k54t7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:51:17.078910 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.078878 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7111e954-0f50-496c-9b61-a5944f6bade8-istio-token" (OuterVolumeSpecName: "istio-token") pod "7111e954-0f50-496c-9b61-a5944f6bade8" (UID: "7111e954-0f50-496c-9b61-a5944f6bade8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:51:17.078910 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.078878 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7111e954-0f50-496c-9b61-a5944f6bade8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7111e954-0f50-496c-9b61-a5944f6bade8" (UID: "7111e954-0f50-496c-9b61-a5944f6bade8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:51:17.178284 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.178258 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7111e954-0f50-496c-9b61-a5944f6bade8-istio-envoy\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:17.178284 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.178279 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7111e954-0f50-496c-9b61-a5944f6bade8-istio-token\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:17.178284 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.178288 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7111e954-0f50-496c-9b61-a5944f6bade8-istio-podinfo\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:17.178478 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.178298 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k54t7\" (UniqueName: \"kubernetes.io/projected/7111e954-0f50-496c-9b61-a5944f6bade8-kube-api-access-k54t7\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:17.768873 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.768843 2572 generic.go:356] "Generic (PLEG): container finished" podID="7111e954-0f50-496c-9b61-a5944f6bade8" containerID="5952f135f82d1611ebf25b965a63d60eb70a1f9fa7f7ca36ceb90bd321c15a9d" exitCode=0 Aug 04 18:51:17.769054 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.768892 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" Aug 04 18:51:17.769054 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.768893 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" event={"ID":"7111e954-0f50-496c-9b61-a5944f6bade8","Type":"ContainerDied","Data":"5952f135f82d1611ebf25b965a63d60eb70a1f9fa7f7ca36ceb90bd321c15a9d"} Aug 04 18:51:17.769054 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.769026 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm" event={"ID":"7111e954-0f50-496c-9b61-a5944f6bade8","Type":"ContainerDied","Data":"3b0b5763762aefc205a5ab10a5a235aa3db0091788e21390a66c5c287702597b"} Aug 04 18:51:17.769158 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.769059 2572 scope.go:117] "RemoveContainer" containerID="5952f135f82d1611ebf25b965a63d60eb70a1f9fa7f7ca36ceb90bd321c15a9d" Aug 04 18:51:17.778403 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.778388 2572 scope.go:117] "RemoveContainer" containerID="a1fab6740f92f6eff8baaa3f52d48ff4fc730a726eeec5dd286e094f81156b3e" Aug 04 18:51:17.786152 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.786137 2572 scope.go:117] "RemoveContainer" containerID="5952f135f82d1611ebf25b965a63d60eb70a1f9fa7f7ca36ceb90bd321c15a9d" Aug 04 18:51:17.786392 ip-10-0-137-173 kubenswrapper[2572]: E0804 18:51:17.786373 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5952f135f82d1611ebf25b965a63d60eb70a1f9fa7f7ca36ceb90bd321c15a9d\": container with ID starting with 5952f135f82d1611ebf25b965a63d60eb70a1f9fa7f7ca36ceb90bd321c15a9d not found: ID does not exist" containerID="5952f135f82d1611ebf25b965a63d60eb70a1f9fa7f7ca36ceb90bd321c15a9d" Aug 04 18:51:17.786440 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.786399 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5952f135f82d1611ebf25b965a63d60eb70a1f9fa7f7ca36ceb90bd321c15a9d"} err="failed to get container status \"5952f135f82d1611ebf25b965a63d60eb70a1f9fa7f7ca36ceb90bd321c15a9d\": rpc error: code = NotFound desc = could not find container \"5952f135f82d1611ebf25b965a63d60eb70a1f9fa7f7ca36ceb90bd321c15a9d\": container with ID starting with 5952f135f82d1611ebf25b965a63d60eb70a1f9fa7f7ca36ceb90bd321c15a9d not found: ID does not exist" Aug 04 18:51:17.786440 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.786413 2572 scope.go:117] "RemoveContainer" containerID="a1fab6740f92f6eff8baaa3f52d48ff4fc730a726eeec5dd286e094f81156b3e" Aug 04 18:51:17.786605 ip-10-0-137-173 kubenswrapper[2572]: E0804 18:51:17.786590 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1fab6740f92f6eff8baaa3f52d48ff4fc730a726eeec5dd286e094f81156b3e\": container with ID starting with a1fab6740f92f6eff8baaa3f52d48ff4fc730a726eeec5dd286e094f81156b3e not found: ID does not exist" containerID="a1fab6740f92f6eff8baaa3f52d48ff4fc730a726eeec5dd286e094f81156b3e" Aug 04 18:51:17.786645 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.786610 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1fab6740f92f6eff8baaa3f52d48ff4fc730a726eeec5dd286e094f81156b3e"} err="failed to get container status \"a1fab6740f92f6eff8baaa3f52d48ff4fc730a726eeec5dd286e094f81156b3e\": rpc error: code = NotFound desc = could not find container \"a1fab6740f92f6eff8baaa3f52d48ff4fc730a726eeec5dd286e094f81156b3e\": container with ID starting with a1fab6740f92f6eff8baaa3f52d48ff4fc730a726eeec5dd286e094f81156b3e not found: ID does not exist" Aug 04 18:51:17.795472 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.795448 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm"] Aug 04 18:51:17.800047 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:17.800026 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6a6d9f-openshift-default-774545b4b6-d8bzm"] Aug 04 18:51:18.821358 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:18.821325 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7111e954-0f50-496c-9b61-a5944f6bade8" path="/var/lib/kubelet/pods/7111e954-0f50-496c-9b61-a5944f6bade8/volumes" Aug 04 18:51:19.874006 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:19.873968 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:20.097434 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:20.097397 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-071def-fnsp7"] Aug 04 18:51:20.100102 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:20.100078 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-071def-fnsp7"] Aug 04 18:51:20.821075 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:20.821044 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4fa775b-b7d7-4ea4-9e34-5a77d2266434" path="/var/lib/kubelet/pods/c4fa775b-b7d7-4ea4-9e34-5a77d2266434/volumes" Aug 04 18:51:22.559441 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:22.559392 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:27.135458 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:27.135416 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:27.207189 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:27.207151 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:27.396715 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:27.396614 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx"] Aug 04 18:51:27.396929 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:27.396882 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" podUID="1974889b-0585-4331-bcfd-025b966f10dd" containerName="istio-proxy" containerID="cri-o://d8c9f609054d16b2c92beb6361fdfe34f76da1c76a5da7d2b04ad28fc9ccf8fe" gracePeriod=30 Aug 04 18:51:27.413615 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:27.413584 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:27.671268 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:27.671148 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:27.818628 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:27.818600 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c8685-9ms5f"] Aug 04 18:51:27.818965 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:27.818952 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7111e954-0f50-496c-9b61-a5944f6bade8" containerName="istio-proxy" Aug 04 18:51:27.819003 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:27.818968 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7111e954-0f50-496c-9b61-a5944f6bade8" containerName="istio-proxy" Aug 04 18:51:27.819003 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:27.818984 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7111e954-0f50-496c-9b61-a5944f6bade8" containerName="istio-proxy" Aug 04 18:51:27.819003 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:27.818989 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7111e954-0f50-496c-9b61-a5944f6bade8" containerName="istio-proxy" Aug 04 18:51:27.819003 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:27.818998 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4fa775b-b7d7-4ea4-9e34-5a77d2266434" containerName="revoke-keys" Aug 04 18:51:27.819144 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:27.819004 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4fa775b-b7d7-4ea4-9e34-5a77d2266434" containerName="revoke-keys" Aug 04 18:51:27.819144 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:27.819062 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4fa775b-b7d7-4ea4-9e34-5a77d2266434" containerName="revoke-keys" Aug 04 18:51:27.819144 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:27.819071 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7111e954-0f50-496c-9b61-a5944f6bade8" containerName="istio-proxy" Aug 04 18:51:27.823623 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:27.823608 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c8685-9ms5f" Aug 04 18:51:27.826541 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:27.826520 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 18:51:27.826651 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:27.826594 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pnbq7\"" Aug 04 18:51:27.830191 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:27.830171 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c8685-9ms5f"] Aug 04 18:51:27.877328 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:27.877298 2572 patch_prober.go:28] interesting pod/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 18:51:27.877446 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:27.877381 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" podUID="1974889b-0585-4331-bcfd-025b966f10dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 18:51:27.968755 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:27.968686 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7812664b-e229-4038-b43b-e3842dfc231f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-5c8685-9ms5f\" (UID: \"7812664b-e229-4038-b43b-e3842dfc231f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c8685-9ms5f" Aug 04 18:51:28.069403 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:28.069370 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7812664b-e229-4038-b43b-e3842dfc231f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-5c8685-9ms5f\" (UID: \"7812664b-e229-4038-b43b-e3842dfc231f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c8685-9ms5f" Aug 04 18:51:28.070029 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:28.070008 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7812664b-e229-4038-b43b-e3842dfc231f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-5c8685-9ms5f\" (UID: \"7812664b-e229-4038-b43b-e3842dfc231f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c8685-9ms5f" Aug 04 18:51:28.134973 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:28.134945 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c8685-9ms5f" Aug 04 18:51:28.272290 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:51:28.272253 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7812664b_e229_4038_b43b_e3842dfc231f.slice/crio-ccbcfc53f12be520eb22eaaa9f2f4b15cc61013770fa6d22eae5de12a9f021b4 WatchSource:0}: Error finding container ccbcfc53f12be520eb22eaaa9f2f4b15cc61013770fa6d22eae5de12a9f021b4: Status 404 returned error can't find the container with id ccbcfc53f12be520eb22eaaa9f2f4b15cc61013770fa6d22eae5de12a9f021b4 Aug 04 18:51:28.287693 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:28.287646 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c8685-9ms5f"] Aug 04 18:51:28.823226 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:28.823190 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c8685-9ms5f" event={"ID":"7812664b-e229-4038-b43b-e3842dfc231f","Type":"ContainerStarted","Data":"596bf432d2c498916ab4a8836af78552a1ba8e96451140e639396a839a46ccfd"} Aug 04 18:51:28.823415 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:28.823250 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c8685-9ms5f" event={"ID":"7812664b-e229-4038-b43b-e3842dfc231f","Type":"ContainerStarted","Data":"ccbcfc53f12be520eb22eaaa9f2f4b15cc61013770fa6d22eae5de12a9f021b4"} Aug 04 18:51:28.841031 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:28.840985 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c8685-9ms5f" podStartSLOduration=1.840967668 podStartE2EDuration="1.840967668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:51:28.840152312 +0000 UTC m=+3544.530843367" watchObservedRunningTime="2026-08-04 18:51:28.840967668 +0000 UTC m=+3544.531658754" Aug 04 18:51:29.830560 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:29.830526 2572 generic.go:356] "Generic (PLEG): container finished" podID="7812664b-e229-4038-b43b-e3842dfc231f" containerID="596bf432d2c498916ab4a8836af78552a1ba8e96451140e639396a839a46ccfd" exitCode=0 Aug 04 18:51:29.830932 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:29.830616 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c8685-9ms5f" event={"ID":"7812664b-e229-4038-b43b-e3842dfc231f","Type":"ContainerDied","Data":"596bf432d2c498916ab4a8836af78552a1ba8e96451140e639396a839a46ccfd"} Aug 04 18:51:30.968043 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:30.968019 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c8685-9ms5f" Aug 04 18:51:31.097868 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:31.097799 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7812664b-e229-4038-b43b-e3842dfc231f-maas-api-service-ca\") pod \"7812664b-e229-4038-b43b-e3842dfc231f\" (UID: \"7812664b-e229-4038-b43b-e3842dfc231f\") " Aug 04 18:51:31.098151 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:31.098131 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7812664b-e229-4038-b43b-e3842dfc231f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7812664b-e229-4038-b43b-e3842dfc231f" (UID: "7812664b-e229-4038-b43b-e3842dfc231f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:51:31.199323 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:31.199294 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7812664b-e229-4038-b43b-e3842dfc231f-maas-api-service-ca\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:31.841280 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:31.841244 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c8685-9ms5f" Aug 04 18:51:31.841441 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:31.841273 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c8685-9ms5f" event={"ID":"7812664b-e229-4038-b43b-e3842dfc231f","Type":"ContainerDied","Data":"ccbcfc53f12be520eb22eaaa9f2f4b15cc61013770fa6d22eae5de12a9f021b4"} Aug 04 18:51:31.841441 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:31.841333 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccbcfc53f12be520eb22eaaa9f2f4b15cc61013770fa6d22eae5de12a9f021b4" Aug 04 18:51:32.364202 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.364167 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:32.656401 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.656379 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:51:32.821284 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.821257 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9t52q\" (UniqueName: \"kubernetes.io/projected/1974889b-0585-4331-bcfd-025b966f10dd-kube-api-access-9t52q\") pod \"1974889b-0585-4331-bcfd-025b966f10dd\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " Aug 04 18:51:32.821453 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.821291 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-credential-socket\") pod \"1974889b-0585-4331-bcfd-025b966f10dd\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " Aug 04 18:51:32.821453 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.821355 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-istio-data\") pod \"1974889b-0585-4331-bcfd-025b966f10dd\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " Aug 04 18:51:32.821453 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.821375 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-istio-envoy\") pod \"1974889b-0585-4331-bcfd-025b966f10dd\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " Aug 04 18:51:32.821453 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.821394 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1974889b-0585-4331-bcfd-025b966f10dd-istiod-ca-cert\") pod \"1974889b-0585-4331-bcfd-025b966f10dd\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " Aug 04 18:51:32.821453 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.821420 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-workload-socket\") pod \"1974889b-0585-4331-bcfd-025b966f10dd\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " Aug 04 18:51:32.821453 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.821452 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-workload-certs\") pod \"1974889b-0585-4331-bcfd-025b966f10dd\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " Aug 04 18:51:32.821785 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.821494 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1974889b-0585-4331-bcfd-025b966f10dd-istio-token\") pod \"1974889b-0585-4331-bcfd-025b966f10dd\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " Aug 04 18:51:32.821785 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.821548 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1974889b-0585-4331-bcfd-025b966f10dd-istio-podinfo\") pod \"1974889b-0585-4331-bcfd-025b966f10dd\" (UID: \"1974889b-0585-4331-bcfd-025b966f10dd\") " Aug 04 18:51:32.821785 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.821755 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1974889b-0585-4331-bcfd-025b966f10dd" (UID: "1974889b-0585-4331-bcfd-025b966f10dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:51:32.821935 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.821776 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1974889b-0585-4331-bcfd-025b966f10dd" (UID: "1974889b-0585-4331-bcfd-025b966f10dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:51:32.821935 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.821877 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-istio-data" (OuterVolumeSpecName: "istio-data") pod "1974889b-0585-4331-bcfd-025b966f10dd" (UID: "1974889b-0585-4331-bcfd-025b966f10dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:51:32.822267 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.821949 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1974889b-0585-4331-bcfd-025b966f10dd" (UID: "1974889b-0585-4331-bcfd-025b966f10dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:51:32.822267 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.821974 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1974889b-0585-4331-bcfd-025b966f10dd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1974889b-0585-4331-bcfd-025b966f10dd" (UID: "1974889b-0585-4331-bcfd-025b966f10dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:51:32.823711 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.823659 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1974889b-0585-4331-bcfd-025b966f10dd-kube-api-access-9t52q" (OuterVolumeSpecName: "kube-api-access-9t52q") pod "1974889b-0585-4331-bcfd-025b966f10dd" (UID: "1974889b-0585-4331-bcfd-025b966f10dd"). InnerVolumeSpecName "kube-api-access-9t52q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:51:32.823961 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.823943 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1974889b-0585-4331-bcfd-025b966f10dd-istio-token" (OuterVolumeSpecName: "istio-token") pod "1974889b-0585-4331-bcfd-025b966f10dd" (UID: "1974889b-0585-4331-bcfd-025b966f10dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:51:32.824205 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.824175 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1974889b-0585-4331-bcfd-025b966f10dd" (UID: "1974889b-0585-4331-bcfd-025b966f10dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:51:32.824302 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.824212 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1974889b-0585-4331-bcfd-025b966f10dd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1974889b-0585-4331-bcfd-025b966f10dd" (UID: "1974889b-0585-4331-bcfd-025b966f10dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:51:32.847761 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.847735 2572 generic.go:356] "Generic (PLEG): container finished" podID="1974889b-0585-4331-bcfd-025b966f10dd" containerID="d8c9f609054d16b2c92beb6361fdfe34f76da1c76a5da7d2b04ad28fc9ccf8fe" exitCode=0 Aug 04 18:51:32.847877 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.847789 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" Aug 04 18:51:32.847877 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.847812 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" event={"ID":"1974889b-0585-4331-bcfd-025b966f10dd","Type":"ContainerDied","Data":"d8c9f609054d16b2c92beb6361fdfe34f76da1c76a5da7d2b04ad28fc9ccf8fe"} Aug 04 18:51:32.847877 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.847861 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx" event={"ID":"1974889b-0585-4331-bcfd-025b966f10dd","Type":"ContainerDied","Data":"8d74481230b798e1914d25b694c4ddf7100b40dd17482f9e124d058744fa2aee"} Aug 04 18:51:32.848042 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.847882 2572 scope.go:117] "RemoveContainer" containerID="d8c9f609054d16b2c92beb6361fdfe34f76da1c76a5da7d2b04ad28fc9ccf8fe" Aug 04 18:51:32.856641 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.856624 2572 scope.go:117] "RemoveContainer" containerID="ef26607f163f27a558d0fb2a321dfa378cf6e88c9e34df847b0e4aaadede01a0" Aug 04 18:51:32.864187 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.864172 2572 scope.go:117] "RemoveContainer" containerID="d8c9f609054d16b2c92beb6361fdfe34f76da1c76a5da7d2b04ad28fc9ccf8fe" Aug 04 18:51:32.864432 ip-10-0-137-173 kubenswrapper[2572]: E0804 18:51:32.864414 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8c9f609054d16b2c92beb6361fdfe34f76da1c76a5da7d2b04ad28fc9ccf8fe\": container with ID starting with d8c9f609054d16b2c92beb6361fdfe34f76da1c76a5da7d2b04ad28fc9ccf8fe not found: ID does not exist" containerID="d8c9f609054d16b2c92beb6361fdfe34f76da1c76a5da7d2b04ad28fc9ccf8fe" Aug 04 18:51:32.864500 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.864441 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c9f609054d16b2c92beb6361fdfe34f76da1c76a5da7d2b04ad28fc9ccf8fe"} err="failed to get container status \"d8c9f609054d16b2c92beb6361fdfe34f76da1c76a5da7d2b04ad28fc9ccf8fe\": rpc error: code = NotFound desc = could not find container \"d8c9f609054d16b2c92beb6361fdfe34f76da1c76a5da7d2b04ad28fc9ccf8fe\": container with ID starting with d8c9f609054d16b2c92beb6361fdfe34f76da1c76a5da7d2b04ad28fc9ccf8fe not found: ID does not exist" Aug 04 18:51:32.864500 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.864457 2572 scope.go:117] "RemoveContainer" containerID="ef26607f163f27a558d0fb2a321dfa378cf6e88c9e34df847b0e4aaadede01a0" Aug 04 18:51:32.864697 ip-10-0-137-173 kubenswrapper[2572]: E0804 18:51:32.864677 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef26607f163f27a558d0fb2a321dfa378cf6e88c9e34df847b0e4aaadede01a0\": container with ID starting with ef26607f163f27a558d0fb2a321dfa378cf6e88c9e34df847b0e4aaadede01a0 not found: ID does not exist" containerID="ef26607f163f27a558d0fb2a321dfa378cf6e88c9e34df847b0e4aaadede01a0" Aug 04 18:51:32.864748 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.864701 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef26607f163f27a558d0fb2a321dfa378cf6e88c9e34df847b0e4aaadede01a0"} err="failed to get container status \"ef26607f163f27a558d0fb2a321dfa378cf6e88c9e34df847b0e4aaadede01a0\": rpc error: code = NotFound desc = could not find container \"ef26607f163f27a558d0fb2a321dfa378cf6e88c9e34df847b0e4aaadede01a0\": container with ID starting with ef26607f163f27a558d0fb2a321dfa378cf6e88c9e34df847b0e4aaadede01a0 not found: ID does not exist" Aug 04 18:51:32.869975 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.869953 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx"] Aug 04 18:51:32.874291 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.874272 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-071def-openshift-default-787cb7bb54-cjxkx"] Aug 04 18:51:32.922862 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.922810 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1974889b-0585-4331-bcfd-025b966f10dd-istiod-ca-cert\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:32.922862 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.922830 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-workload-socket\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:32.922862 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.922841 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-workload-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:32.922862 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.922850 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1974889b-0585-4331-bcfd-025b966f10dd-istio-token\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:32.922862 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.922857 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1974889b-0585-4331-bcfd-025b966f10dd-istio-podinfo\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:32.923058 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.922866 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t52q\" (UniqueName: \"kubernetes.io/projected/1974889b-0585-4331-bcfd-025b966f10dd-kube-api-access-9t52q\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:32.923058 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.922875 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-credential-socket\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:32.923058 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.922883 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-istio-data\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:32.923058 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:32.922891 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1974889b-0585-4331-bcfd-025b966f10dd-istio-envoy\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:34.824874 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:34.822880 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1974889b-0585-4331-bcfd-025b966f10dd" path="/var/lib/kubelet/pods/1974889b-0585-4331-bcfd-025b966f10dd/volumes" Aug 04 18:51:35.204853 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:35.204756 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:35.205342 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:35.205322 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c8685-29764485-xkclk"] Aug 04 18:51:35.208721 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:35.208697 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c8685-29764485-xkclk"] Aug 04 18:51:35.290569 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:35.290527 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:35.389019 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:35.388981 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:35.451042 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:35.450997 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:35.561410 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:35.561383 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c8685-9ms5f"] Aug 04 18:51:35.564391 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:35.564362 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c8685-9ms5f"] Aug 04 18:51:35.941536 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:35.941453 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:36.821508 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:36.821465 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7812664b-e229-4038-b43b-e3842dfc231f" path="/var/lib/kubelet/pods/7812664b-e229-4038-b43b-e3842dfc231f/volumes" Aug 04 18:51:36.821874 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:36.821853 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82d01363-e20c-4b00-a039-b418289cbb0f" path="/var/lib/kubelet/pods/82d01363-e20c-4b00-a039-b418289cbb0f/volumes" Aug 04 18:51:42.929129 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:42.929101 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:43.028678 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.028633 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:43.116354 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.116322 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9"] Aug 04 18:51:43.116607 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.116580 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" podUID="3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" containerName="istio-proxy" containerID="cri-o://b68a104185eb56cc3a11cdabd60bad2421f6e82ab8672630bd81f2d74c9606b5" gracePeriod=30 Aug 04 18:51:43.136862 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.136818 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:43.213397 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.213308 2572 patch_prober.go:28] interesting pod/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 18:51:43.213397 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.213366 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" podUID="3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 18:51:43.396571 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.396538 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:43.582695 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.582624 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc18f1-c4mjq"] Aug 04 18:51:43.583552 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.583515 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7812664b-e229-4038-b43b-e3842dfc231f" containerName="revoke-keys" Aug 04 18:51:43.583552 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.583543 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7812664b-e229-4038-b43b-e3842dfc231f" containerName="revoke-keys" Aug 04 18:51:43.583783 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.583574 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1974889b-0585-4331-bcfd-025b966f10dd" containerName="istio-proxy" Aug 04 18:51:43.583783 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.583583 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1974889b-0585-4331-bcfd-025b966f10dd" containerName="istio-proxy" Aug 04 18:51:43.583783 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.583638 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1974889b-0585-4331-bcfd-025b966f10dd" containerName="istio-proxy" Aug 04 18:51:43.583783 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.583646 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1974889b-0585-4331-bcfd-025b966f10dd" containerName="istio-proxy" Aug 04 18:51:43.583971 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.583900 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7812664b-e229-4038-b43b-e3842dfc231f" containerName="revoke-keys" Aug 04 18:51:43.583971 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.583924 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1974889b-0585-4331-bcfd-025b966f10dd" containerName="istio-proxy" Aug 04 18:51:43.583971 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.583934 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1974889b-0585-4331-bcfd-025b966f10dd" containerName="istio-proxy" Aug 04 18:51:43.583971 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.583953 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7111e954-0f50-496c-9b61-a5944f6bade8" containerName="istio-proxy" Aug 04 18:51:43.588889 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.588863 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc18f1-c4mjq" Aug 04 18:51:43.591459 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.591432 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 18:51:43.591616 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.591466 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pnbq7\"" Aug 04 18:51:43.593534 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.593502 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc18f1-c4mjq"] Aug 04 18:51:43.614437 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.614411 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7198716-9c48-4b9d-8e50-1fce5043d6c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-bc18f1-c4mjq\" (UID: \"b7198716-9c48-4b9d-8e50-1fce5043d6c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc18f1-c4mjq" Aug 04 18:51:43.715424 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.715391 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7198716-9c48-4b9d-8e50-1fce5043d6c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-bc18f1-c4mjq\" (UID: \"b7198716-9c48-4b9d-8e50-1fce5043d6c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc18f1-c4mjq" Aug 04 18:51:43.715979 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.715963 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7198716-9c48-4b9d-8e50-1fce5043d6c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-bc18f1-c4mjq\" (UID: \"b7198716-9c48-4b9d-8e50-1fce5043d6c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc18f1-c4mjq" Aug 04 18:51:43.902330 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:43.902250 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc18f1-c4mjq" Aug 04 18:51:44.039518 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:44.039493 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc18f1-c4mjq"] Aug 04 18:51:44.041760 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:51:44.041735 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7198716_9c48_4b9d_8e50_1fce5043d6c3.slice/crio-b9c35685f1374fb6622f1fd0b4932cc51da8c87939818cdd1c7a99759d9b4f4b WatchSource:0}: Error finding container b9c35685f1374fb6622f1fd0b4932cc51da8c87939818cdd1c7a99759d9b4f4b: Status 404 returned error can't find the container with id b9c35685f1374fb6622f1fd0b4932cc51da8c87939818cdd1c7a99759d9b4f4b Aug 04 18:51:44.904961 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:44.904929 2572 generic.go:356] "Generic (PLEG): container finished" podID="b7198716-9c48-4b9d-8e50-1fce5043d6c3" containerID="a5dae907d3557c0d799e4b28f01a0e304d22fefb3ba89184173177a504c965ee" exitCode=0 Aug 04 18:51:44.905111 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:44.905016 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc18f1-c4mjq" event={"ID":"b7198716-9c48-4b9d-8e50-1fce5043d6c3","Type":"ContainerDied","Data":"a5dae907d3557c0d799e4b28f01a0e304d22fefb3ba89184173177a504c965ee"} Aug 04 18:51:44.905111 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:44.905075 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc18f1-c4mjq" event={"ID":"b7198716-9c48-4b9d-8e50-1fce5043d6c3","Type":"ContainerStarted","Data":"b9c35685f1374fb6622f1fd0b4932cc51da8c87939818cdd1c7a99759d9b4f4b"} Aug 04 18:51:46.037865 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:46.037840 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc18f1-c4mjq" Aug 04 18:51:46.131723 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:46.131695 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7198716-9c48-4b9d-8e50-1fce5043d6c3-maas-api-service-ca\") pod \"b7198716-9c48-4b9d-8e50-1fce5043d6c3\" (UID: \"b7198716-9c48-4b9d-8e50-1fce5043d6c3\") " Aug 04 18:51:46.132050 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:46.132028 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7198716-9c48-4b9d-8e50-1fce5043d6c3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b7198716-9c48-4b9d-8e50-1fce5043d6c3" (UID: "b7198716-9c48-4b9d-8e50-1fce5043d6c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:51:46.232728 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:46.232630 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7198716-9c48-4b9d-8e50-1fce5043d6c3-maas-api-service-ca\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:46.915657 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:46.915631 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc18f1-c4mjq" Aug 04 18:51:46.915657 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:46.915653 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc18f1-c4mjq" event={"ID":"b7198716-9c48-4b9d-8e50-1fce5043d6c3","Type":"ContainerDied","Data":"b9c35685f1374fb6622f1fd0b4932cc51da8c87939818cdd1c7a99759d9b4f4b"} Aug 04 18:51:46.915849 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:46.915713 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9c35685f1374fb6622f1fd0b4932cc51da8c87939818cdd1c7a99759d9b4f4b" Aug 04 18:51:48.319974 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.319939 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc18f1-29764485-8qzst"] Aug 04 18:51:48.322836 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.322813 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-bc18f1-29764485-8qzst"] Aug 04 18:51:48.375084 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.375059 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:51:48.449960 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.449879 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istio-data\") pod \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " Aug 04 18:51:48.449960 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.449919 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-workload-socket\") pod \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " Aug 04 18:51:48.449960 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.449952 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istio-token\") pod \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " Aug 04 18:51:48.450240 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.449971 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istio-podinfo\") pod \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " Aug 04 18:51:48.450240 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.449990 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rt2wk\" (UniqueName: \"kubernetes.io/projected/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-kube-api-access-rt2wk\") pod \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " Aug 04 18:51:48.450240 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.450062 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istio-envoy\") pod \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " Aug 04 18:51:48.450240 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.450087 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-workload-certs\") pod \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " Aug 04 18:51:48.450240 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.450119 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-credential-socket\") pod \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " Aug 04 18:51:48.450240 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.450154 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istiod-ca-cert\") pod \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\" (UID: \"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d\") " Aug 04 18:51:48.450552 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.450237 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" (UID: "3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:51:48.450552 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.450247 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istio-data" (OuterVolumeSpecName: "istio-data") pod "3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" (UID: "3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:51:48.450552 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.450390 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" (UID: "3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:51:48.450552 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.450455 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" (UID: "3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:51:48.450552 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.450493 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-workload-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:48.450552 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.450505 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istio-data\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:48.450552 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.450513 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-workload-socket\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:48.451123 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.450568 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" (UID: "3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 18:51:48.452398 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.452369 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istio-token" (OuterVolumeSpecName: "istio-token") pod "3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" (UID: "3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:51:48.452398 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.452388 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" (UID: "3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 18:51:48.452600 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.452503 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-kube-api-access-rt2wk" (OuterVolumeSpecName: "kube-api-access-rt2wk") pod "3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" (UID: "3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d"). InnerVolumeSpecName "kube-api-access-rt2wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 18:51:48.452793 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.452775 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" (UID: "3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 18:51:48.551502 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.551473 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-credential-socket\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:48.551502 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.551501 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istiod-ca-cert\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:48.551702 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.551511 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istio-token\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:48.551702 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.551519 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istio-podinfo\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:48.551702 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.551527 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt2wk\" (UniqueName: \"kubernetes.io/projected/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-kube-api-access-rt2wk\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:48.551702 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.551535 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d-istio-envoy\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Aug 04 18:51:48.557823 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.557802 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:48.615829 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.615797 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:48.674070 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.674039 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc18f1-c4mjq"] Aug 04 18:51:48.677804 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.677771 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-bc18f1-c4mjq"] Aug 04 18:51:48.821391 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.821354 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7198716-9c48-4b9d-8e50-1fce5043d6c3" path="/var/lib/kubelet/pods/b7198716-9c48-4b9d-8e50-1fce5043d6c3/volumes" Aug 04 18:51:48.821753 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.821730 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f49fe828-3ed9-4b3f-ae6e-a4e1b8ba9f8a" path="/var/lib/kubelet/pods/f49fe828-3ed9-4b3f-ae6e-a4e1b8ba9f8a/volumes" Aug 04 18:51:48.926851 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.926765 2572 generic.go:356] "Generic (PLEG): container finished" podID="3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" containerID="b68a104185eb56cc3a11cdabd60bad2421f6e82ab8672630bd81f2d74c9606b5" exitCode=0 Aug 04 18:51:48.926851 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.926811 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" Aug 04 18:51:48.926851 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.926849 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" event={"ID":"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d","Type":"ContainerDied","Data":"b68a104185eb56cc3a11cdabd60bad2421f6e82ab8672630bd81f2d74c9606b5"} Aug 04 18:51:48.927125 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.926902 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9" event={"ID":"3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d","Type":"ContainerDied","Data":"5821c15f9ff3be2e9185b2cf07290b0546ae531da5ab12ba9197e4232fb91625"} Aug 04 18:51:48.927125 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.926927 2572 scope.go:117] "RemoveContainer" containerID="b68a104185eb56cc3a11cdabd60bad2421f6e82ab8672630bd81f2d74c9606b5" Aug 04 18:51:48.935584 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.935564 2572 scope.go:117] "RemoveContainer" containerID="356acf44522306aa3d55a8b6e95d07452f39aaf693dbaee13fb07f585d7726c1" Aug 04 18:51:48.943359 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.943343 2572 scope.go:117] "RemoveContainer" containerID="b68a104185eb56cc3a11cdabd60bad2421f6e82ab8672630bd81f2d74c9606b5" Aug 04 18:51:48.943606 ip-10-0-137-173 kubenswrapper[2572]: E0804 18:51:48.943588 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68a104185eb56cc3a11cdabd60bad2421f6e82ab8672630bd81f2d74c9606b5\": container with ID starting with b68a104185eb56cc3a11cdabd60bad2421f6e82ab8672630bd81f2d74c9606b5 not found: ID does not exist" containerID="b68a104185eb56cc3a11cdabd60bad2421f6e82ab8672630bd81f2d74c9606b5" Aug 04 18:51:48.943672 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.943615 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68a104185eb56cc3a11cdabd60bad2421f6e82ab8672630bd81f2d74c9606b5"} err="failed to get container status \"b68a104185eb56cc3a11cdabd60bad2421f6e82ab8672630bd81f2d74c9606b5\": rpc error: code = NotFound desc = could not find container \"b68a104185eb56cc3a11cdabd60bad2421f6e82ab8672630bd81f2d74c9606b5\": container with ID starting with b68a104185eb56cc3a11cdabd60bad2421f6e82ab8672630bd81f2d74c9606b5 not found: ID does not exist" Aug 04 18:51:48.943672 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.943631 2572 scope.go:117] "RemoveContainer" containerID="356acf44522306aa3d55a8b6e95d07452f39aaf693dbaee13fb07f585d7726c1" Aug 04 18:51:48.943893 ip-10-0-137-173 kubenswrapper[2572]: E0804 18:51:48.943878 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"356acf44522306aa3d55a8b6e95d07452f39aaf693dbaee13fb07f585d7726c1\": container with ID starting with 356acf44522306aa3d55a8b6e95d07452f39aaf693dbaee13fb07f585d7726c1 not found: ID does not exist" containerID="356acf44522306aa3d55a8b6e95d07452f39aaf693dbaee13fb07f585d7726c1" Aug 04 18:51:48.943931 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.943897 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"356acf44522306aa3d55a8b6e95d07452f39aaf693dbaee13fb07f585d7726c1"} err="failed to get container status \"356acf44522306aa3d55a8b6e95d07452f39aaf693dbaee13fb07f585d7726c1\": rpc error: code = NotFound desc = could not find container \"356acf44522306aa3d55a8b6e95d07452f39aaf693dbaee13fb07f585d7726c1\": container with ID starting with 356acf44522306aa3d55a8b6e95d07452f39aaf693dbaee13fb07f585d7726c1 not found: ID does not exist" Aug 04 18:51:48.947282 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.947260 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9"] Aug 04 18:51:48.951678 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:48.951645 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-5c8685-openshift-default-6b4cc8f4b4-cz6k9"] Aug 04 18:51:49.169041 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:49.169010 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:50.822536 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:50.822505 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" path="/var/lib/kubelet/pods/3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d/volumes" Aug 04 18:51:53.182756 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:53.182723 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:55.320251 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:55.320220 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:55.383471 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:55.383438 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:55.625237 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:55.625147 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:51:56.055888 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:51:56.055853 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:02.781205 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:02.781171 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz_5fdc80a1-1524-40c4-8e22-edfb5b0167ce/manager/0.log" Aug 04 18:52:02.789363 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:02.789339 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz_5fdc80a1-1524-40c4-8e22-edfb5b0167ce/copy-manifests/0.log" Aug 04 18:52:03.426375 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:03.426342 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29764455-45sjp_2263abad-9b33-4666-8e3e-22b76fe5bae7/cleanup/0.log" Aug 04 18:52:03.544618 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:03.544578 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29764470-s2wgq_a1c0b408-8066-4900-b5e0-75df9bf2ebf6/cleanup/0.log" Aug 04 18:52:03.664231 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:03.664205 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29764485-8h6kz_ab4fe340-35fe-4c8f-af78-45d769926a01/cleanup/0.log" Aug 04 18:52:05.869826 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:05.869801 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-k5r8f_e191e8f4-ac16-4f98-8c33-0b9a6b313992/discovery/0.log" Aug 04 18:52:05.993081 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:05.993046 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-798d66f497-gw6lc_4907cbad-4833-473c-9b6b-b41272f3c90a/kube-auth-proxy/0.log" Aug 04 18:52:07.613609 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:07.613574 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-sqcds_2c7cd685-f67d-4c9f-89bf-4cda37dda6f6/main/0.log" Aug 04 18:52:08.598563 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:08.598532 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:13.622818 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:13.622783 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:13.695844 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:13.695809 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" limit=1024 Aug 04 18:52:16.542041 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:16.541960 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xbhvg_096b3bbb-ead1-451f-9f3a-e5a4add161ad/global-pull-secret-syncer/0.log" Aug 04 18:52:16.635561 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:16.635518 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-64c4r_b3f65ebe-8341-406d-bc45-a6e176fccd97/konnectivity-agent/0.log" Aug 04 18:52:16.877286 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:16.877257 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-173.ec2.internal_37a95ca805868321653a87ed06a2c559/haproxy/0.log" Aug 04 18:52:23.148323 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:23.148295 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_de29157d-9827-4529-ae90-6f457f90d7db/alertmanager/0.log" Aug 04 18:52:23.186060 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:23.186029 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_de29157d-9827-4529-ae90-6f457f90d7db/config-reloader/0.log" Aug 04 18:52:23.232102 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:23.232080 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_de29157d-9827-4529-ae90-6f457f90d7db/kube-rbac-proxy-web/0.log" Aug 04 18:52:23.274301 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:23.274278 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_de29157d-9827-4529-ae90-6f457f90d7db/kube-rbac-proxy/0.log" Aug 04 18:52:23.305542 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:23.305519 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_de29157d-9827-4529-ae90-6f457f90d7db/kube-rbac-proxy-metric/0.log" Aug 04 18:52:23.335769 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:23.335749 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_de29157d-9827-4529-ae90-6f457f90d7db/prom-label-proxy/0.log" Aug 04 18:52:23.369124 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:23.369072 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_de29157d-9827-4529-ae90-6f457f90d7db/init-config-reloader/0.log" Aug 04 18:52:23.687605 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:23.687583 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-78w5w_6a5eec29-205a-4c1a-8a0e-c156b7c6c334/monitoring-plugin/0.log" Aug 04 18:52:23.990675 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:23.990582 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hnrz6_cfca7ce7-765d-4d8a-8ea1-a8f9c5931466/node-exporter/0.log" Aug 04 18:52:24.036016 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:24.035994 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hnrz6_cfca7ce7-765d-4d8a-8ea1-a8f9c5931466/kube-rbac-proxy/0.log" Aug 04 18:52:24.067732 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:24.067707 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hnrz6_cfca7ce7-765d-4d8a-8ea1-a8f9c5931466/init-textfile/0.log" Aug 04 18:52:24.504129 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:24.504105 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_42aefbf1-664e-4883-9277-18bfc9174296/prometheus/0.log" Aug 04 18:52:24.531963 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:24.531942 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_42aefbf1-664e-4883-9277-18bfc9174296/config-reloader/0.log" Aug 04 18:52:24.560777 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:24.560753 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_42aefbf1-664e-4883-9277-18bfc9174296/thanos-sidecar/0.log" Aug 04 18:52:24.590077 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:24.590060 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_42aefbf1-664e-4883-9277-18bfc9174296/kube-rbac-proxy-web/0.log" Aug 04 18:52:24.618328 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:24.618306 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_42aefbf1-664e-4883-9277-18bfc9174296/kube-rbac-proxy/0.log" Aug 04 18:52:24.649815 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:24.649738 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_42aefbf1-664e-4883-9277-18bfc9174296/kube-rbac-proxy-thanos/0.log" Aug 04 18:52:24.674405 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:24.674384 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_42aefbf1-664e-4883-9277-18bfc9174296/init-config-reloader/0.log" Aug 04 18:52:24.710061 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:24.710041 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-hbqtm_5ef17428-3a91-41ab-b3a3-525d6eaffa6f/prometheus-operator/0.log" Aug 04 18:52:24.738283 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:24.738267 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-hbqtm_5ef17428-3a91-41ab-b3a3-525d6eaffa6f/kube-rbac-proxy/0.log" Aug 04 18:52:24.940789 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:24.940726 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c8d9fcf-f6b4w_d2e3e902-9da0-498d-87ff-3e9040b86081/thanos-query/0.log" Aug 04 18:52:24.968297 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:24.968275 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c8d9fcf-f6b4w_d2e3e902-9da0-498d-87ff-3e9040b86081/kube-rbac-proxy-web/0.log" Aug 04 18:52:24.994017 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:24.993993 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c8d9fcf-f6b4w_d2e3e902-9da0-498d-87ff-3e9040b86081/kube-rbac-proxy/0.log" Aug 04 18:52:25.023155 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:25.023128 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c8d9fcf-f6b4w_d2e3e902-9da0-498d-87ff-3e9040b86081/prom-label-proxy/0.log" Aug 04 18:52:25.051745 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:25.051716 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c8d9fcf-f6b4w_d2e3e902-9da0-498d-87ff-3e9040b86081/kube-rbac-proxy-rules/0.log" Aug 04 18:52:25.084210 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:25.084189 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c8d9fcf-f6b4w_d2e3e902-9da0-498d-87ff-3e9040b86081/kube-rbac-proxy-metrics/0.log" Aug 04 18:52:25.246077 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:25.246001 2572 scope.go:117] "RemoveContainer" containerID="ddc7fd424f71d8f3d27044b6eef81130ae8af4ca37bb8cec5451a2e521df8b63" Aug 04 18:52:25.254240 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:25.254220 2572 scope.go:117] "RemoveContainer" containerID="fe98b7b201ce1b007d7103ee225333e5662ae3eb85a92222faf3b2a536bf2c74" Aug 04 18:52:26.411331 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.411298 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb"] Aug 04 18:52:26.411704 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.411633 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" containerName="istio-proxy" Aug 04 18:52:26.411704 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.411644 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" containerName="istio-proxy" Aug 04 18:52:26.411704 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.411651 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" containerName="istio-proxy" Aug 04 18:52:26.411704 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.411656 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" containerName="istio-proxy" Aug 04 18:52:26.411704 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.411679 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7198716-9c48-4b9d-8e50-1fce5043d6c3" containerName="revoke-keys" Aug 04 18:52:26.411704 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.411685 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7198716-9c48-4b9d-8e50-1fce5043d6c3" containerName="revoke-keys" Aug 04 18:52:26.411898 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.411747 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7198716-9c48-4b9d-8e50-1fce5043d6c3" containerName="revoke-keys" Aug 04 18:52:26.411898 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.411760 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" containerName="istio-proxy" Aug 04 18:52:26.411898 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.411766 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c8a5bd9-0ad2-442b-94d9-8d4cc35e403d" containerName="istio-proxy" Aug 04 18:52:26.416519 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.416496 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb" Aug 04 18:52:26.422044 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.422021 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-v4pcl\"/\"default-dockercfg-hmk57\"" Aug 04 18:52:26.422142 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.422030 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v4pcl\"/\"kube-root-ca.crt\"" Aug 04 18:52:26.422910 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.422892 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v4pcl\"/\"openshift-service-ca.crt\"" Aug 04 18:52:26.430770 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.430739 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb"] Aug 04 18:52:26.579570 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.579540 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/840c8264-efbf-42a6-886c-15bfff71c2f0-lib-modules\") pod \"perf-node-gather-daemonset-v4jsb\" (UID: \"840c8264-efbf-42a6-886c-15bfff71c2f0\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb" Aug 04 18:52:26.579759 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.579585 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/840c8264-efbf-42a6-886c-15bfff71c2f0-proc\") pod \"perf-node-gather-daemonset-v4jsb\" (UID: \"840c8264-efbf-42a6-886c-15bfff71c2f0\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb" Aug 04 18:52:26.579759 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.579683 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/840c8264-efbf-42a6-886c-15bfff71c2f0-podres\") pod \"perf-node-gather-daemonset-v4jsb\" (UID: \"840c8264-efbf-42a6-886c-15bfff71c2f0\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb" Aug 04 18:52:26.579759 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.579714 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/840c8264-efbf-42a6-886c-15bfff71c2f0-sys\") pod \"perf-node-gather-daemonset-v4jsb\" (UID: \"840c8264-efbf-42a6-886c-15bfff71c2f0\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb" Aug 04 18:52:26.579759 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.579739 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5gs8\" (UniqueName: \"kubernetes.io/projected/840c8264-efbf-42a6-886c-15bfff71c2f0-kube-api-access-v5gs8\") pod \"perf-node-gather-daemonset-v4jsb\" (UID: \"840c8264-efbf-42a6-886c-15bfff71c2f0\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb" Aug 04 18:52:26.680739 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.680619 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/840c8264-efbf-42a6-886c-15bfff71c2f0-proc\") pod \"perf-node-gather-daemonset-v4jsb\" (UID: \"840c8264-efbf-42a6-886c-15bfff71c2f0\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb" Aug 04 18:52:26.680739 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.680722 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/840c8264-efbf-42a6-886c-15bfff71c2f0-podres\") pod \"perf-node-gather-daemonset-v4jsb\" (UID: \"840c8264-efbf-42a6-886c-15bfff71c2f0\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb" Aug 04 18:52:26.680739 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.680740 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/840c8264-efbf-42a6-886c-15bfff71c2f0-proc\") pod \"perf-node-gather-daemonset-v4jsb\" (UID: \"840c8264-efbf-42a6-886c-15bfff71c2f0\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb" Aug 04 18:52:26.681000 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.680747 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/840c8264-efbf-42a6-886c-15bfff71c2f0-sys\") pod \"perf-node-gather-daemonset-v4jsb\" (UID: \"840c8264-efbf-42a6-886c-15bfff71c2f0\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb" Aug 04 18:52:26.681000 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.680791 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/840c8264-efbf-42a6-886c-15bfff71c2f0-sys\") pod \"perf-node-gather-daemonset-v4jsb\" (UID: \"840c8264-efbf-42a6-886c-15bfff71c2f0\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb" Aug 04 18:52:26.681000 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.680795 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5gs8\" (UniqueName: \"kubernetes.io/projected/840c8264-efbf-42a6-886c-15bfff71c2f0-kube-api-access-v5gs8\") pod \"perf-node-gather-daemonset-v4jsb\" (UID: \"840c8264-efbf-42a6-886c-15bfff71c2f0\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb" Aug 04 18:52:26.681000 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.680837 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/840c8264-efbf-42a6-886c-15bfff71c2f0-lib-modules\") pod \"perf-node-gather-daemonset-v4jsb\" (UID: \"840c8264-efbf-42a6-886c-15bfff71c2f0\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb" Aug 04 18:52:26.681000 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.680902 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/840c8264-efbf-42a6-886c-15bfff71c2f0-podres\") pod \"perf-node-gather-daemonset-v4jsb\" (UID: \"840c8264-efbf-42a6-886c-15bfff71c2f0\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb" Aug 04 18:52:26.681000 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.680948 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/840c8264-efbf-42a6-886c-15bfff71c2f0-lib-modules\") pod \"perf-node-gather-daemonset-v4jsb\" (UID: \"840c8264-efbf-42a6-886c-15bfff71c2f0\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb" Aug 04 18:52:26.690460 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.690443 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5gs8\" (UniqueName: \"kubernetes.io/projected/840c8264-efbf-42a6-886c-15bfff71c2f0-kube-api-access-v5gs8\") pod \"perf-node-gather-daemonset-v4jsb\" (UID: \"840c8264-efbf-42a6-886c-15bfff71c2f0\") " pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb" Aug 04 18:52:26.726545 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.726521 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb" Aug 04 18:52:26.856918 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:26.856886 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb"] Aug 04 18:52:26.859860 ip-10-0-137-173 kubenswrapper[2572]: W0804 18:52:26.859828 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod840c8264_efbf_42a6_886c_15bfff71c2f0.slice/crio-9e0b168305b6b6d03b6fd04cc812b122fa798488754ededa9d7e08850d5b04d0 WatchSource:0}: Error finding container 9e0b168305b6b6d03b6fd04cc812b122fa798488754ededa9d7e08850d5b04d0: Status 404 returned error can't find the container with id 9e0b168305b6b6d03b6fd04cc812b122fa798488754ededa9d7e08850d5b04d0 Aug 04 18:52:27.105256 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:27.105214 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb" event={"ID":"840c8264-efbf-42a6-886c-15bfff71c2f0","Type":"ContainerStarted","Data":"1503e7d072332a1554ce66242b8073e5e87b22b574de4efff5428300874478ae"} Aug 04 18:52:27.105458 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:27.105279 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb" event={"ID":"840c8264-efbf-42a6-886c-15bfff71c2f0","Type":"ContainerStarted","Data":"9e0b168305b6b6d03b6fd04cc812b122fa798488754ededa9d7e08850d5b04d0"} Aug 04 18:52:27.105458 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:27.105415 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb" Aug 04 18:52:27.125749 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:27.125704 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb" podStartSLOduration=1.125690736 podStartE2EDuration="1.125690736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:52:27.123318358 +0000 UTC m=+3602.814009408" watchObservedRunningTime="2026-08-04 18:52:27.125690736 +0000 UTC m=+3602.816381787" Aug 04 18:52:27.682293 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:27.682264 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-77c974ffcf-gf5hr_48ff3a32-3414-4512-a8d8-7918e3ae955a/download-server/0.log" Aug 04 18:52:29.576891 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:29.576838 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bj5tc_6dad9eb4-a215-485c-961f-412ef770df62/dns/0.log" Aug 04 18:52:29.606840 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:29.606817 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bj5tc_6dad9eb4-a215-485c-961f-412ef770df62/kube-rbac-proxy/0.log" Aug 04 18:52:29.691814 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:29.691792 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4dgxz_c600bdf9-3cef-457a-bb0b-704c5856c077/dns-node-resolver/0.log" Aug 04 18:52:30.377654 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:30.377629 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6ns66_8d005c33-5c1e-4ec8-990d-ee4d27743422/node-ca/0.log" Aug 04 18:52:31.587508 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:31.587474 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-k5r8f_e191e8f4-ac16-4f98-8c33-0b9a6b313992/discovery/0.log" Aug 04 18:52:31.620652 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:31.620625 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-798d66f497-gw6lc_4907cbad-4833-473c-9b6b-b41272f3c90a/kube-auth-proxy/0.log" Aug 04 18:52:32.768372 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:32.768303 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wjdss_3505d3f2-19bc-4965-ae4d-3dc9bff42667/serve-healthcheck-canary/0.log" Aug 04 18:52:33.119612 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:33.119582 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-v4pcl/perf-node-gather-daemonset-v4jsb" Aug 04 18:52:33.505738 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:33.505652 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vxvb6_ea1559b9-3848-4974-9c46-eac22ac768e3/kube-rbac-proxy/0.log" Aug 04 18:52:33.532651 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:33.532627 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vxvb6_ea1559b9-3848-4974-9c46-eac22ac768e3/exporter/0.log" Aug 04 18:52:33.562158 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:33.562138 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vxvb6_ea1559b9-3848-4974-9c46-eac22ac768e3/extractor/0.log" Aug 04 18:52:36.638863 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:36.638831 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz_5fdc80a1-1524-40c4-8e22-edfb5b0167ce/manager/0.log" Aug 04 18:52:36.680389 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:36.680365 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fd6db6bcb-b7jtz_5fdc80a1-1524-40c4-8e22-edfb5b0167ce/copy-manifests/0.log" Aug 04 18:52:37.379341 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:37.379308 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-k5r8f_e191e8f4-ac16-4f98-8c33-0b9a6b313992/discovery/0.log" Aug 04 18:52:37.390918 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:37.390900 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-798d66f497-gw6lc_4907cbad-4833-473c-9b6b-b41272f3c90a/kube-auth-proxy/0.log" Aug 04 18:52:38.136459 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:38.136431 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-55bf5c9f58-bfs22_bb7ec098-1dd9-4212-bbf2-d69562f92b97/manager/0.log" Aug 04 18:52:39.166866 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:39.166786 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-k5r8f_e191e8f4-ac16-4f98-8c33-0b9a6b313992/discovery/0.log" Aug 04 18:52:39.180325 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:39.180299 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-798d66f497-gw6lc_4907cbad-4833-473c-9b6b-b41272f3c90a/kube-auth-proxy/0.log" Aug 04 18:52:45.015219 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:45.015152 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t2nnw_a1dbfaf6-9a2f-4146-807c-418181c922c1/kube-multus-additional-cni-plugins/0.log" Aug 04 18:52:45.038736 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:45.038705 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t2nnw_a1dbfaf6-9a2f-4146-807c-418181c922c1/egress-router-binary-copy/0.log" Aug 04 18:52:45.068700 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:45.068659 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t2nnw_a1dbfaf6-9a2f-4146-807c-418181c922c1/cni-plugins/0.log" Aug 04 18:52:45.094134 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:45.094109 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t2nnw_a1dbfaf6-9a2f-4146-807c-418181c922c1/bond-cni-plugin/0.log" Aug 04 18:52:45.118888 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:45.118866 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t2nnw_a1dbfaf6-9a2f-4146-807c-418181c922c1/routeoverride-cni/0.log" Aug 04 18:52:45.141712 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:45.141685 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t2nnw_a1dbfaf6-9a2f-4146-807c-418181c922c1/whereabouts-cni-bincopy/0.log" Aug 04 18:52:45.166733 ip-10-0-137-173 kubenswrapper[2572]: I0804 18:52:45.166707 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t2nnw_a1dbfaf6-9a2f-4146-807c-418181c922c1/whereabouts-cni/0.log"